Building Windows 8

An inside look from the Windows engineering team

Acting on file management feedback

Mon, 30 Jan 2012 20:00:00 GMT -

As we approach our next public milestone, we will begin to circle back to topics we covered in the blog and talk about the changes we’ve made to the product since the Developer Preview. As we've said often, we read the comments, newsgroup discussions, and reviews that have been written about Windows 8 and track the feedback carefully. We listen to this feedback by taking into account the source of the feedback and factoring in the intended audience for features as well as trying to reconcile conflicting feedback (no matter how many thumbs up votes there might be, we can promise that, for any design worth discussing, there are conflicting and equally valid points of view). Of course, we always consider the engineering feasibility of any changes we make—compatibility, security, performance, and so on.

Ilana Smith, a lead program manager on the Engineering System team, authored this post.

--Steven

---

We previously published three blog posts that discussed the new file management experience in Windows 8: one about the new copy experience, one that detailed the design process we went through for the new conflict experience and one about the changes to Windows Explorer, including the introduction of the ribbon.

Those posts prompted great discussion and we read the approximately 2200 comments you left. This was wonderful feedback for us, and, along with information from our other feedback channels, we incorporated it into our design process.

Summarizing blog post comments

As we prepare for the beta, we thought we would update you on some of the key issues, and the changes you should expect to see.

Conflict: identifying duplicate files during conflict resolution

In Windows 8, we have a new experience for selecting the right file when file name collisions are encountered during a copy or move.

L. Brown said:

A compare button to show if files are equal in the "Choose" dialog would be really great!

Frequently, the reason two files have the same name is because they’re copies. Making a choice between two identical files is usually pointless – it’s unnecessary for a copy operation, and often unnecessary for a move operation. We looked at several methods of identifying duplicate files and decided that checking the file name, file size and date modified attributes was the most effective approach. They can be used to identify the vast majority of duplicate files quickly, efficiently, and with good backward compatibility compared to other methods like file hashes.

In the beta, we’ve added a new option to the detailed conflict resolution dialog. By checking the box in the bottom left of the dialog, you can filter out all files that match on name, size (down to the byte) and time (down to the granularity of the file system timestamp: 2 seconds for FAT, 100 nanoseconds for NTFS). The system will skip copying or moving these files. This functionality adds no additional time to the operation, works both locally and across networks, and on all types of systems and storage.

We’ll skip copying files with the same name, date, and size

This check box is deselected by default (to ensure users opt into the changed behavior), but it persists once you select it.

Copy: system changes

JL asked:

You know when you start a big copy job and realize that you are doing it over the wireless so you grab a network cable and plug it in? Does the file copy know to utilize the faster connection now?

If both sides of the copy operation are on Windows 8 machines, yes, it will be able to take advantage of the increased network throughput on the fly, thanks to advancements in the Server Message Block (SMB) protocol to support multiple channels.

Tobi asked:

Will it be possible to pause the copy operation and resume it after reboot/sleep/hibernate?

In the beta, when a system sleeps or hibernates, the copy operation will automatically pause, and when the machine wakes, you can choose to resume the copy by clicking the depressed pause button. (We decided not to have copies automatically resume on wake, as the system environment may have changed significantly in the interim and we do not want to cause an error.)

Copy: handling confirmations and interrupts

gawicks asked:

Please please display all the copy 'error dialogs' after the copying has completed so I don't have to sit in front of the machine all the time.

We have two types of user interaction that can occur during a copy job - we break these into two groups, “confirmations” and ”interrupts.” Confirmations like “Are you sure you want to permanently delete this file?” need to be completed before the copy operation can start. Interrupts are issues that the system encounters while copying, things like “File not found,” “File in use,” and file name conflicts.

The system presents all confirmations before it starts to move or copy files. While copying, any interrupt issues are queued and presented once the system has completed all the work it can. In the beta, we’ve made improvements in how confirmations are presented, making sure they don’t get lost amongst existing running copies.

Explorer: navigation pane scrolling issue

xpclient said

Please fix the infamous Windows 7 navigation pane scrolling bug.

(See this Microsoft Answers thread for more information.)

We fixed it! As of the beta, it’s gone.

Explorer: respect picture orientation metadata

Raf asked

Will you support *lossless* picture rotation?

In Windows 7 and 8, JPEG rotation is lossless when both image dimensions are divisible by 16 (standard image sizes).

Additionally, Explorer now respects EXIF orientation information for JPEG images. If your camera sets this value accurately, you will rarely need to correct orientation. Look for a future blog post where we will discuss this in more detail.

Images in Windows 7 Explorer

Images in Windows 8 Explorer

 

Explorer: Overlay changes to improve performance

In Windows 8, we continue to prioritize great performance. We pay close attention to milliseconds of lag and look for reductions. In Explorer, we found an opportunity for improvement in delays caused by icon overlays.

In Windows 7, we have a padlock icon overlay to indicate a private file. (You might recall that, due to the increase in shared files, it had superseded the ”palm up” overlay for shared files.) We recently found that checking for these overlays was adding about 120 milliseconds to our Explorer library launch tests. This might not seem like much, but we consider this a big delay.

Overlays have limitations – they can only show a single state, add a lot of visual noise, and can be confusing. The padlock overlay has been removed; this information is conveyed better by the “Sharing status” column.

This column has these advantages:

• Performance: The column is hidden by default, so the delay is incurred only when you opt into showing this information.
• Tri-state: This column has three values: Shared, Not shared, and Private, so you get more detail than you would from the icon overlay.
• Sorting/filtering: You can sort and filter the sharing status property, providing more powerful file management capabilities.

Explorer: pin to Start

On Marina’s post about the Start screen, Boots112233 said:

Half of the items in my Windows 7 Start Menu are shortcuts to folders and one is to a file […] How can I do this in Windows 8 if the start screen won't allow shortcuts for folders?

In the beta, you can now easily pin your favorite folders to Start, and take advantage of the rich customization functionality that we built into it to arrange the folders into groups and into any order you want.

"Pin to Start" from the Windows Explorer ribbon

Additionally, just as in Windows 7, you can pin shortcuts to executables to Start directly from Windows Explorer, which can be very useful for applications that don’t add themselves to the Start screen by default.

Documents folder, a custom executable, and Event Viewer pinned to Start

Explorer: PowerShell

Jamie Thomson said:

Really liking the "Open Command prompt" option in the File menu however I prefer to use PowerShell so would like an "Open PowerShell prompt" option too.

We agree, and so we added this as well. It is worth noting that there are sometimes conflicting points of view on whether advanced things should be in the GUI or in PowerShell, and how front and center they should be. We are always balancing the complexity of too many options and too many ways to do things. As you can see, there is no right answer, so we'll continue to balance these complex choices.

Windows PowerShell buttons in Windows Explorer

These menu items launch the PowerShell console. The PowerShell ISE continues to be available from the Edit command on a PowerShell file.

Explorer: ribbon changes

We had expected the introduction of the ribbon to Explorer to spur conversation, and it is fair to say the voluminous response was in line with our expectations. It’s exciting to work on something that brings so many different perspectives.

There were many reactions, and as we expected, there is a set of people who have an entirely negative reaction to the affordance and have been telling us about it in no uncertain terms. Our view is that we do need to move the user interface forward and accept that a vocal set of customers are just not happy with the direction we're going. When looked at broadly, that is balanced out by a majority of people who are happy and more productive with the changes. We remind folks that there are third-party tools available (likely the tools being used by this set of people), that provide a number of different interface paradigms. We do embrace the notion that third-party tools play an important part in the Windows experience.

That said, we’ve internalized your feedback, experimented with and tested various approaches, and used our co-workers as test subjects, in addition to the formal testing as you would expect. You’ll see three major changes in the ribbon in the beta.

Ribbon minimized by default: With the ribbon maximized in the Developer Preview, we’ve been able to learn a lot about how people interact with it, which has enabled us to tweak and fine-tune it. With the beta, we will be making a major change that brings Explorer in line with our design principles for Windows 8. As in our copy dialogs, Task Manager, and Metro style experiences, we will be reducing distractions and trusting users to discover functionality on their own, by minimizing the ribbon by default.

Windows Explorer ribbon minimized by default

We’ve tested this change for a while now, and the results have been heartening. This is data from internal usage at Microsoft, which we know not to be representative of broad audiences, but is generally representative of the folks like you that engage in the dialog on the blog.

This data shows that our very tech-savvy users are generally fine with either setting, but that our heavier Explorer users are our ribbon maximizers. For lighter file browsing scenarios, we can provide a UI with reduced distractions, and still trust that users who want to really exercise Explorer functionality will maximize and leverage the ribbon.

Visible hotkeys: Our telemetry data has shown us that for users who actively choose to minimize the ribbon, their strong preference is to use hotkeys. The ribbon provides new ways to access functionality via the keyboard with keytips (those floating cues that pop up when you hit Alt), but traditional shortcut keys like Ctrl+V remain the most efficient method. We love shortcut keys (internally, their usage gets up over 85% of all Explorer commands issued), so we want to help more people discover them.

For the beta release, we’ve added hotkey information to the tooltips of relevant buttons.

“New folder” tooltip shows the keyboard shortcut

User setting roaming: We want to make sure you only need configure your Explorer options once. If you maximize your ribbon, and add Undo and Map Network Drive in your Quick Access Toolbar, we want your Explorer to look like that every time.

For the beta release, we’ve added Explorer settings to the attributes that are roamed to your other Windows 8 PCs. In the “Sync your settings” UI, this shows up under “Other Windows settings.” (For more information about roaming user settings, take a look at Katie’s post.)

Syncing Explorer settings across PCs

We really appreciate all your feedback on our previous posts. We believe it has contributed directly to an improved file management experience for Windows 8.

--Ilana Smith

Supporting sensors in Windows 8

Tue, 24 Jan 2012 18:00:00 GMT -

Recent advances in sensor technology are catalysts for the acceleration and evolution of user experiences on PCs. The ability to react to changes in ambient light, motion, human proximity, and location are becoming common and essential elements of the computing experience. Even something simple—like an ambient light sensor to adjust display brightness in a room with changing light—is potentially a basic scenario for desktop PCs. Of course, we also want to make sure you have full control over the use of these peripherals, since we know that different sensors leave open opportunities for risk or abuse that some folks might not be comfortable with. This post looks at the details of supporting sensors in Windows 8 and was authored by Gavin Gear, a PM on the Device Connectivity team.
--Steven

---

The first thing we explored about sensors was how Windows 8 should use them at the system level, to adapt the PC to the environment while preserving battery life.

Adaptive brightness

The first system feature was automatic display brightness control, or what we call “adaptive brightness.” This was a feature that we first introduced in Windows 7 using ambient light sensors (ALS), and is targeted at mobile form factors like slates, convertibles, and laptops. With today’s display panels supporting brightness levels at approximately twice the intensity of what was common just a few years ago, this feature is more important than ever. By dynamically controlling screen brightness based on changing ambient light conditions, we can optimize the level of reading comfort, and save battery life when the screen is dimmed in darker environments.

 

A tablet PC in harsh outdoor lighting with adaptive brightness (left), and without (right)

You can see here that adaptive brightness helps you see content on the screen more clearly, since the screen automatically gets brighter when the tablet enters a bright environment. And for those of you who use your desktop PCs in a sunny room, you know this same thing can happen at different times of the day in different seasons.

Automatic screen rotation

Many smartphones and other mobile devices have established the expectation that when you rotate the device, the graphic display will also rotate and adapt to the new orientation (including adapting to aspect ratio changes). Data from an accelerometer allows the device to determine its basic orientation. By automatically rotating the screen, people can use their devices (primarily slates and convertibles) in a more natural and intuitive way, without needing to manually rotate the screen with software controls or hardware buttons.

 

Windows 8 Start screen in landscape and portrait orientations

Developer support for sensors

Beyond figuring out the basics for how a Windows 8 system might use sensors, we also needed to think about how apps might use sensors. We looked at a variety of examples of sensor-enabled apps including games, commercial applications, tools, and utilities, to help us determine which scenarios to support.

First on the list was the ability for apps to understand motion and screen rotation. This requires an accelerometer – a device that can be used to measure the force due to gravity, and the motion of the device itself. But most scenarios require more than just an understanding of motion and gravity. Orientation is also an important requirement for many applications. To enable a PC to understand orientation we needed to integrate the functionality of a compass.

Supporting a compass would at minimum require a 3D accelerometer (which measures acceleration on three axes) and a 3D magnetometer (which measures magnetic field strengths on 3 axes). This combination of sensors is called a 6-axis motion and orientation sensing system, and can support a basic tilt-compensated compass, screen rotation, and certain casual game apps like a labyrinth style game. However, in our testing and prototyping, we found the 6-axis motion sensing system has two key drawbacks: sporadic compass inaccuracy, and a lack of the responsiveness required by 3D interactive games.

Recently, a new type of sensor has started to emerge on phone platforms – the gyro sensor. Gyro sensors measure angular speed, typically along 3 axes. You can also use the data from gyro sensors to increase the responsiveness and accuracy of 3D motion-sensing systems. A gyro sensor is very sensitive, but it lacks any form of orientation reference (such as gravity or north heading).

This diagram shows how gyro data is represented as a set of three rotations along the three primary axes for the device:

Initially, some thought that the need for such sensors was scoped to very few apps, such as specialized games. But the more we examined the 3D motion and orientation sensing problem, the more we realized that applications are much more immersive and attractive if they react to the kind of motion humans can easily understand, such as shakes, twists, and rotations in multiple dimensions. With these kinds of sensors it would certainly be possible to build very immersive 3D games, but it would also enable lots of other apps to more naturally respond to input from a variety of motions, including mapping and navigation applications, measuring utilities, interactive (between two machines) applications, and simple apps like casual games.

Engineering challenges

We started our exploration into motion apps by prototyping some 3D experiences. The first challenge was to map the physical orientation of the device directly to a virtual 3D environment in the app. We decided to model a simple augmented reality experience by emulating a tablet as a window into a virtual world. The concept was fairly simple: when you move the device while looking at the screen, the virtual environment (the inside of a room) would appear to stay stationary.

Initially, we tried an experiment using the accelerometer to map up and down movement of the device to up and down movement of the 3D environment in response. When you hold the device still, the scene should remain stable. When you tilt the device, the view should tilt up or down. Right away we encountered an issue: “noise” in the data from the accelerometer sensor was causing jittery movement of the 3D environment even when the device was held stationary. We were able to see this noise clearly by capturing accelerometer data and charting it.

Without noise, the lines on the chart would be straight, with no vertical deviation. The conventional way to remove such noise is to apply a low-pass filter to the raw data stream. When we implemented this mitigation in our prototype, the resultant motion was smooth and stable (jitter-free). But the low-pass filter introduced another problem: the app lost responsiveness and felt sluggish when responding to motion. We needed a way to compensate for this jitter without reducing responsiveness.

The next experiment was to provide the ability to “look left” and “look right” in our virtual 3D environment app. We used a 6-axis compass solution (3D accelerometer + 3D magnetometer) to support this type of movement. Although this kind of worked, the movement was not consistent due to the general instability of the 6-axis compass. It was also challenging to blend the up-and-down movement with the left-and-right movement.

From these experiments it was clear that this combination of sensors could not provide the fluid and responsive experience we wanted. The accelerometer sensor was not providing clean data, and could not be used alone to determine device orientation. The magnetometer was slow to update and was susceptible to electromagnetic interference (think of a compass needle that sticks in one position occasionally). We had yet to experiment with the gyro sensors, but because gyros could only determine rotational speed, it wasn’t clear how they could help.

Creating “sensor fusion”

But further experimentation demonstrated that using all three sensors together could solve the problem. It turns out that an accelerometer, magnetometer, and a gyro can complement each-other’s weaknesses, effectively filling in gaps in data and data responsiveness. Using a combination of these sensors it is possible to create a better, more responsive, and more fluid experience than the sensors can provide individually. Combining the input of multiple sensors to produce better overall results is a process we call sensor fusion.

Essentially, sensor fusion is a case where the whole is greater than the sum of the parts. A typical sensor fusion system uses a 3D accelerometer, a 3D magnetometer, and a 3D gyro to create a combined “9-axis sensor fusion” system. To understand how this system works, let’s take a look at the inputs and outputs.

9-axis sensor fusion system

This diagram shows two types of outputs: pass-through outputs in which the sensor data is passed directly to an application, and sensor fusion outputs in which the sensor data is synthesized into more powerful data types.

Some applications can use pass-through sensor data directly. This data can be used at “face value” for a variety of scenarios. One such scenario is an app that implements a pedometer to count your steps as you walk. The graph below shows the output of the accelerometer for a person walking with a tablet PC. This graph clearly shows it is possible to detect every step the person took.

 

But, as our experiments revealed, many applications can’t effectively use the raw sensor data. Some of these applications include:

• Compass apps
• Enhanced navigation and augmented reality apps
• Casual games
• 3D gaming apps

Here’s a screenshot from a 3D game sample:

3D first-person shooter game (shown at //Build/)

These applications need to use sensor fusion data in order to support the features they implement. The “magic” of sensor fusion is to mathematically combine the data from all three sensors to produce more sophisticated outputs, including a tilt-compensated compass, an inclinometer (exposing yaw, pitch, and roll), and more advanced representations of device orientation. With this kind of data, more sophisticated apps can produce fast, fluid, and responsive reactions to natural motions.

By integrating a sensor fusion solution, Windows 8 provides a complete solution for the full range of applications. Sensor fusion in Windows solves the problems of jittery movement and jerky transitions, reduces data integrity issues, and provides data that allows a seamless representation of full device motion in 3D space (without any awkward transitions).

Working with hardware partners

While designing a sensor fusion solution for Windows, we also needed to help hardware designers to take advantage of this solution by partnering with them early. Designing a sensor fusion system is relatively easy if you’re designing a single device. But Windows runs on many kinds of PCs in many form factors, using hardware components from many different manufacturers. We needed to provide a solution that enabled the entire ecosystem of Windows hardware partners to participate.

The first step was to provide a baseline of performance for sensor packages that would work with Windows’ sensor fusion solution. Using Windows certification guidelines, we provided specifications for sensor performance. To help hardware companies verify that their solutions were compatible with Windows, we built a number of tests, which we provide with the Windows Certification kit.

Reducing the cost of developing and supporting drivers was another challenge. In order to make it simpler for sensor hardware manufacturers and PC makers, we wrote a single Microsoft-supplied driver that would work with all Windows-compatible sensor packages connected over USB and even lower power busses like I2C. This sensor class driver enables hardware companies to innovate with sensor hardware while ensuring that their hardware can be supported easily with drivers that ship with the Windows operating system.

To help speed adoption of the class driver, Microsoft worked with industry partners to introduce the specification into public standards. In July 2011 the standard for sensors was introduced in the HID (Human Interface Device) specification of the USB-IF (HID spec version 1.12, introduced with review request #39). This standardization enables any sensor company to build a sensor package that is compatible with Windows 8 by following the public standard USB-IF specifications for compliant device firmware. This reduces the time and cost required to integrate sensor hardware with Windows 8 PCs. Other benefits include a lower support cost and more consistent hardware capabilities for Windows 8 PCs that are equipped with sensors.

But beyond standardizing the class driver, we also wanted to optimize the performance of the sensor fusion solution, and minimize its impact on battery life. Each active sensor on a system draws power, and sending data up the stack consumes both memory and CPU time. We helped minimize the power and performance impact for sensor fusion systems running on Windows 8 in two major ways:

1. We architected the sensor fusion interfaces in Windows 8 to enable much of the processing of sensor fusion data to happen at the hardware level. This hardware-level sensor fusion capability means that computationally expensive algorithms don’t have to run on the main CPU, saving power and CPU cycles.

2. We implemented powerful filtering mechanisms that we tied directly to the needs of sensor apps running at any given point in time. This pay-for-play data and event model means that sensor data is only sent up the stack at the rate that apps need that data, and no faster. This results in greatly reduced CPU utilization for sensor data throughput.

Sensors and Metro style apps

To pull all of this together, our final challenge was to make the power and promise of sensor fusion available to those writing Metro style apps. To enable this, we designed a sensor API as part of the new WinRT. Through these APIs, developers can access the power of sensor fusion from any Metro style app. These APIs are clean and simple, and at the same time give developers access to the data needed to support everything from casual games to virtual reality applications. Of course these capabilities are all available as Win32 APIs for game developers or other uses in desktop applications.

The following JavaScript code snippet shows how easy it is to get access to an accelerometer and subscribe to events using the Windows Runtime:

var accelerometer;
accelerometer = Windows.Devices.Sensors.Accelerometer.getDefault();
accelerometer.addEventListener("readingchanged",onAccReadingChanged);

function onAccReadingChanged(e) {
    var accelX = e.reading.accelerationX;
    var accelY = e.reading.accelerationY;
    var accelZ = e.reading.accelerationZ;
}

For more information about support for sensors in the Windows Runtime, please see this //build/ session on using location & sensors in your app.

You may be wondering at this point how you can try out sensor fusion on Windows 8, or even write some apps that use these new capabilities. Developers who attended the //build/ conference in 2011 received the Samsung Windows 8 Developer Preview slate PC, which included a full package of sensors. There were only about 4,000 of those given out, so of course, not everyone had the opportunity to get one. The good news is that the same 9-axis sensor fusion system that was built into the Windows Developer Preview device is now available online for purchase from ST Microelectronics. The “ST Microelectronics eMotion Development Board for Windows 8” (model # STEVAL-MKI119V1) attaches via USB, and works with the HID sensor class driver that’s included in Windows 8. If you’ve downloaded the Developer Preview version of Windows 8 and are itching to try out the sensor experience you should consider getting one of these devices.

ST Microelectronics eMotion Development Board for Windows 8

Now let’s take a look at sensor fusion in action!

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

-- Gavin

Engineering Windows 8 for mobile networks

Fri, 20 Jan 2012 18:30:00 GMT -

In this post, we dig into the details of how we have re-engineered the wireless networking stack to optimize it for both mobile broadband and Wi-Fi networks. We’ve done a ton of work to enable mobile broadband providers to make it easy for you to use 3G and 4G connectivity along with Wi-Fi in Windows 8. In addition to this architectural work, we’ve worked on keeping Windows connected to a network even when in a low-power state (when the screen is off, for example) when running on supporting architectures/PCs. You can learn more about this in the //build/ sessions on connected standby. Billy Anders, a group program manager on our devices and networking team, authored this post.
--Steven

People want similar mobility on their PCs as they get on their smartphones.

It is unlikely that your end goal is just to get connected to the Internet. Instead, connecting to the Internet is a step (or a hurdle) towards what you really want to do, like surf, socialize, or explore, and you would prefer that your PC is connected and ready for you to use whenever you want and wherever you are.

We looked at the fundamentals of wireless connectivity and re-engineered Windows 8 for a mobile and wireless future, going beyond incremental improvements. This is a good example of work that requires new hardware to work in concert with new software in order to realize its full potential.

Simplifying your mobile broadband experience

We knew that if we were to give you true mobility, that Wi-Fi alone would not be enough. Therefore, for Windows 8, we fully developed and integrated mobile broadband (MB) as a first-class connectivity experience within Windows – right alongside Wi-Fi.

We first included mobile broadband in Windows 7, but if you were a mobile broadband user, you likely had a number of hurdles to overcome before connecting with mobile broadband. Yes, you needed the requisite mobile broadband hardware (e.g., mobile broadband dongle or embedded module and SIM) and data plan, but you also needed to locate and install third-party device drivers, and in some cases software, before ever getting your first connection. If the drivers for your device and software from your mobile operator were not available locally, you had to find another connection type (perhaps Wi-Fi) to the Internet to search for software on the websites of the PC maker or mobile operator. This placed a sizable hurdle in front of users trying to connect with mobile broadband, right when they most needed that connection.

We wanted to eliminate the guesswork in locating and installing device drivers for mobile broadband. We did this by working with our mobile operator and mobile broadband hardware partners across the industry, designing a hardware specification that device makers can incorporate into their device hardware. In Windows 8, we developed an in-box mobile broadband class driver that works with all of these devices and eliminates your need for additional device driver software. You just plug in the device and connect. The driver stays up to date via Windows Update, ensuring you have a reliable mobile broadband experience.

The USB Implementers Forum (USB-IF) recently approved the Mobile Broadband Interface Model (MBIM) specification as a standard, and major device makers have already begun adopting this standard into their device designs, including some designed for other operating systems. For more information on the specification, see the USB-IF press release.

Helping you manage your connections and radios

Typically, mobile broadband devices come with radio and connection management software. Device manufacturers, PC manufacturers, and mobile operators all develop, distribute, and support these applications for you to connect to their networks, turn radios on and off, configure connection settings, and get contact information for help and support. Prior to Windows 8, you needed these applications to compensate for functionality not provided natively in Windows. This additional software confused and frustrated users by conflicting with the Windows connection manager, showing different networks, network status, and a separate user interface. Windows 8 eliminates this confusion by providing simple, intuitive, and fully integrated radio and connection management.

The new Windows 8 network settings allow you to turn individual radios on and off (Wi-Fi, mobile broadband, or Bluetooth), as well as disable all radios at once with the new “airplane mode.” Windows 8 provides native radio management to eliminate the conflicts and confusion, and to provide a consistent experience for controlling your radios without the need to install additional software. This is new for PCs even though it has obviously long been available on today’s mobile phones (or Windows Mobile phones, going way back).

You can turn airplane mode on or off in one click

The new wireless network settings in Windows 8 allow you to see and connect to all available MB and Wi-Fi networks from one convenient user interface. We made sure that this interface is consistent and allows you to think less about which network you want to connect. Windows does this by starting with the right default behaviors, and then it gets smarter by learning your network preferences over time.

One of those default behaviors is to prioritize Wi-Fi networks over broadband whenever one of your preferred Wi-Fi networks is available. Wi-Fi networks are typically faster, with lower latency, and have higher data caps (if they are not free). When you connect to a Wi-Fi network, we automatically disconnect you from your mobile broadband network and, when appropriate, power down the mobile broadband device, which also increases battery life. If no preferred Wi-Fi network is available, we automatically reconnect you to your preferred mobile broadband network.

To make sure we connect to the right network when multiple networks are available, Windows maintains an ordered list of your preferred networks based on your explicit connect and disconnect actions, as well as the network type. For example, if you manually disconnect from a network, Windows will no longer automatically connect to that network. If, while connected to one network, you decide to connect to a different network, Windows will move the new network higher in your preferred networks list. Windows automatically learns your preferences in order to manage this list for you.

When you resume from standby, Windows can also reconnect you faster to your preferred Wi-Fi networks by optimizing operations in the networking stack, and providing your network list, connection information, and hints to your Wi-Fi adapter. Now when your PC resumes from standby, your Wi-Fi adapter already has all the information it needs to connect to your preferred Wi-Fi networks. This means you can reconnect your PC to a Wi-Fi network from standby in about a second –oftentimes before your display is even ready. You do not have to do anything special for this – Windows just learns which networks you prefer and manages everything for you. This work was a major part of the architectural work we did in the networking stack and with our hardware partners.

Getting connected to mobile broadband

Even with its broad availability, Wi-Fi by itself does not enable the ubiquitous Internet access that users increasingly want. True mobility requires mobile broadband, which provides connectivity over cellular networks (the same networks as your smartphone). However, just including mobile broadband in Windows 8 was not enough. We also wanted to remove any hurdles to getting you connected to mobile broadband, making it simpler, more intuitive, and more like Wi-Fi.

We made things simpler and more intuitive by fully integrating mobile broadband into Windows 8. When you’re ready to connect to a mobile broadband network, you simply insert your mobile broadband device or SIM card into your Windows 8 PC and we take care of the setup.

If you have a carrier-unlocked mobile broadband device that supports carrier switching (this includes most mobile broadband users outside the US), Windows 8 has native support that allows you to select and connect to any supported carrier from within the Windows UI.

Selecting from available carriers (with supported hardware)

We’ve already talked about how we removed the need to install a driver, or a radio and connection manager. We also automatically identify which mobile operator is associated with your device (or SIM card), brand it in the Windows connection manager with the mobile operator’s logo, configure the PC for connecting to the mobile operator’s network, and download the operator’s mobile broadband app (if they have one) from the Windows Store.

If you purchased and activated a data plan along with your SIM or mobile broadband device, all you need to do is connect to the network and we get out of the way, allowing you to do what you want to do.

Getting connected via mobile broadband using an AT&T SIM card

If you don’t already have a data plan and would like to purchase one, then simply click the “Connect” button for the mobile operator you want, and we automatically direct you to their mobile broadband app or website, where you can select a data plan (for example, a time-based, limit-based, or subscription-based plan). 

AT&T’s new mobile broadband app walks you through purchasing a data plan

After you’ve purchased your plan, your mobile operator provisions your PC over the air for their network, including information about your data plan details and Wi-Fi hotspots.

  Usage details are shown with the connected account

Behind the scenes, Windows identifies the mobile broadband subscriber information, looks up the mobile operator in the new Access Point Name (APN) database, and pre-provisions the system to connect to the operator’s network. Meanwhile, your core connection experience stays the same.

The operator’s mobile broadband app is available via the “View my account” link, or from the app’s tile on the Start screen. Here, you can see how much data you’ve used, pay your bill, manage your account, and get customer support.

AT&T mobile broadband app, account overview

Avoiding “bill shock”

Many of us have read headlines about people receiving surprisingly expensive bills from their mobile operators. The industry has termed this bill shock, and the problem has received enough attention that some governments have begun taking regulatory steps that ask mobile operators to alert their customers when their data usage reaches a certain threshold. Today, mobile operators all have different ways of responding when subscribers exceed their data usage allotment. An operator may block your Internet access, throttle (slow down) your data speed, or simply begin charging you per kilobyte or megabyte. If you are unaware that you are over your data usage limit, you will likely continue using your data plan and rack up additional charges, resulting in shock when you receive your bill.

Prior to Windows 8, we maintained consistent behavior on all types of networks relative to bandwidth usage. With Windows 8, we now take the cost of the network into consideration: we assume that mobile broadband networks have restrictive data caps with higher overage costs (vs. Wi-Fi), and adjust networking behavior with these metered networks accordingly.

As mentioned earlier, we automatically disconnect from mobile broadband and connect you to your preferred Wi-Fi networks whenever they’re available. This reduces your data usage on mobile broadband when possible.

Because many of us use public Wi-Fi, Windows 8 includes support for popular Wi-Fi hotspot authentication types, including WISPr (Wireless Internet Services Provider roaming), EAP-SIM/AKA/AKA Prime (SIM-based authentication), and EAP-TTLS (popular on university campuses). Windows manages the authentication for you when you come within range of a Wi-Fi network that uses one of these methods, so you won’t have to re-authenticate each time (for instance, by going to a web page). This means you get the same automatic behavior at a public Wi-Fi hotspot as you would at home or the office.

On a PC that has both mobile broadband and Wi-Fi, we’ll move you from MB to the less costly Wi-Fi network automatically whenever Wi-Fi is available, again reducing your mobile broadband usage and your potential for bill shock.

Another way we optimize your bandwidth usage is by changing the Windows Update download behavior. For a majority of users, who have turned on automatic updating, Windows Update will defer the background download of all updates until you connect to a non-metered network, such as your home broadband connection. There is one exception, as noted in our earlier post on Windows Update, and that is in the case of a critical security update to fix a worm-like vulnerability (e.g., a Blaster worm). In that case, Windows Update will download the update regardless of the network type. You can always override the deferred download by launching Windows Update and manually initiating the download of updates at a time more convenient to you. Again, you are in full control of your device.

We recognize that most fixed-line broadband plans also have data caps and overage fees. Those data caps are typically much higher than mobile broadband, and therefore we do not change the behavior for these connections. You are always in control and can always mark any wireless network as metered or unmetered by selecting “reduce data usage” in the right-click (or tap and hold) menu for that network.

Marking the Wi-Fi connection as “metered”

We also want Windows applications to behave well on metered networks, so we’ve provided a new set of developer APIs within the ConnectionCost class of the Windows.Networking.Connectivity namespace. If you are an application developer, we encourage you to leverage these APIs and adapt the behavior of your app, such as allowing a low-definition vs. high-definition video stream, or a header-only vs. full-sync of email, depending on the network type. We believe that this adaptive behavior is critical, as it results in actual cost savings for end users. All Metro style apps in the Windows Store must implement these APIs if they use the network.

Even with Windows and other applications behaving smartly on the network, you still may to want to know how much data you have consumed. Windows 8 provides local data usage counters right within the network settings. These counters provide real time local data usage estimates for Wi-Fi and mobile broadband network connections.

Local data usage estimates

The local counters keep track of the amount of data used on each individual network type so you don’t have to. You can reset the counter whenever you want, which may be useful if you want to monitor your usage month-to-month or even within a session. Although you should think of the local data counters as a quick way to determine your usage, they are not a substitute for what mobile operators report as their usage, which may vary slightly, and should be available in the operator’s app.

Another way we help you manage your mobile broadband data usage is by allowing mobile operators to alert you as you approach your bandwidth cap. Some countries have already begun to mandate that operators send messages to subscribers as they approach their bandwidth cap, or once they begin roaming to a different network. The mobile operator sends you an SMS or USSD alert as you approach your bandwidth cap (e.g., 70% used, 85% used, etc.), and the MB operator’s app notifies you and updates its Start screen tile. The following screen shots show what is already available in the Windows Developer Preview (and on the Samsung Preview PC that had an AT&T SIM and plan).

Data usage notification, bottom right.

Data usage information on the mobile operator’s app tile

The Windows 8 task manager provides more granular information if you want to know how much data a particular app has consumed on the network. Here, you can see the approximate active and historical data consumption of any process over metered and non-metered networks. With this information, you can take control by identifying which apps are consuming the most bandwidth and taking action if needed.

Data consumption information in the Windows Task Manager

Here’s a short video that demonstrates some of the new wireless networking features and enhancements in Windows 8.

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

We designed Windows 8 with you—and mobility—in mind. We set out to simplify your experience with getting and staying connected across mobile broadband and Wi-Fi networks, removing hurdles and whenever possible, doing the right things automatically for you.

-- Billy Anders

Building the next generation file system for Windows: ReFS

Mon, 16 Jan 2012 22:00:00 GMT -

We wanted to continue our dialog about data storage by talking about the next generation file system being introduced in Windows 8.  Today, NTFS is the most widely used, advanced, and feature rich file system in broad use. But when you’re reimagining Windows, as we are for Windows 8, we don’t rest on past successes, and so with Windows 8 we are also introducing a newly engineered file system. ReFS, (which stands for Resilient File System), is built on the foundations of NTFS, so it maintains crucial compatibility while at the same time it has been architected and engineered for a new generation of storage technologies and scenarios. In Windows 8, ReFS will be introduced only as part of Windows Server 8, which is the same approach we have used for each and every file system introduction. Of course at the application level, ReFS stored data will be accessible from clients just as NTFS data would be. As you read this, let’s not forget that NTFS is by far the industry’s leading technology for file systems on PCs.

This detailed architectural post was authored by Surendra Verma, a development manager on our Storage and File System team, though, as with every feature, a lot of folks contributed. We have also used the FAQ approach again in this post.
--Steven

PS: Don't forget to track us on @buildwindows8 where we were providing some updates from CES. 

---

In this blog post I’d like to talk about a new file system for Windows. This file system, which we call ReFS, has been designed from the ground up to meet a broad set of customer requirements, both today’s and tomorrow’s, for all the different ways that Windows is deployed.

The key goals of ReFS are:

• Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.
• Verify and auto-correct data. Data can get corrupted due to a number of reasons and therefore must be verified and, when possible, corrected automatically. Metadata must not be written in place to avoid the possibility of “torn writes,” which we will talk about in more detail below.
• Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.
• Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.
• Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.

The key features of ReFS are as follows (note that some of these features are provided in conjunction with Storage Spaces).

• Metadata integrity with checksums
• Integrity streams providing optional user data integrity
• Allocate on write transactional model for robust disk updates (also known as copy on write)
• Large volume, file and directory sizes
• Storage pooling and virtualization makes file system creation and management easy
• Data striping for performance (bandwidth can be managed) and redundancy for fault tolerance
• Disk scrubbing for protection against latent disk errors
• Resiliency to corruptions with "salvage" for maximum volume availability in all cases
• Shared storage pools across machines for additional failure tolerance and load balancing

In addition, ReFS inherits the features and semantics from NTFS including BitLocker encryption, access-control lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks.

And of course, data stored on ReFS is accessible through the same file access APIs on clients that are used on any operating system that can access today’s NTFS volumes.

Key design attributes and features

Our design attributes are closely related to our goals. As we go through these attributes, keep in mind the history of producing file systems used by hundreds of millions of devices scaling from the smallest footprint machines to the largest data centers, from the smallest storage format to the largest multi-spindle format, from solid state storage to the largest drives and storage systems available. Yet at the same time, Windows file systems are accessed by the widest array of application and system software anywhere. ReFS takes that learning and builds on it. We didn’t start from scratch, but reimagined it where it made sense and built on the right parts of NTFS where that made sense. Above all, we are delivering this in a pragmatic manner consistent with the delivery of a major file system—something only Microsoft has done at this scale.

Code reuse and compatibility

When we look at the file system API, this is the area where compatibility is the most critical and technically, the most challenging. Rewriting the code that implements file system semantics would not lead to the right level of compatibility and the issues introduced would be highly dependent on application code, call timing, and hardware. Therefore in building ReFS, we reused the code responsible for implementing the Windows file system semantics. This code implements the file system interface (read, write, open, close, change notification, etc.), maintains in-memory file and volume state, enforces security, and maintains memory caching and synchronization for file data. This reuse ensures a high degree of compatibility with the features of NTFS that we’re carrying forward.

Underneath this reused portion, the NTFS version of the code-base uses a newly architected engine that implements on-disk structures such as the Master File Table (MFT) to represent files and directories. ReFS combines this reused code with a brand-new engine, where a significant portion of the innovation behind ReFS lies. Graphically, it looks like this:

Reliable and scalable on-disk structures

On-disk structures and their manipulation are handled by the on-disk storage engine. This exposes a generic key-value interface, which the layer above leverages to implement files, directories, etc. For its own implementation, the storage engine uses B+ trees exclusively. In fact, we utilize B+ trees as the single common on-disk structure to represent all information on the disk. Trees can be embedded within other trees (a child tree’s root is stored within the row of a parent tree). On the disk, trees can be very large and multi-level or really compact with just a few keys and embedded in another structure. This ensures extreme scalability up and down for all aspects of the file system. Having a single structure significantly simplifies the system and reduces code. The new engine interface includes the notion of “tables” that are enumerable sets of key-value pairs. Most tables have a unique ID (called the object ID) by which they can be referenced. A special object table indexes all such tables in the system.

Now, let’s look at how the common file system abstractions are constructed using tables.

File structures

As shown in the diagram above, directories are represented as tables. Because we implement tables using B+ trees, directories can scale efficiently, becoming very large. Files are implemented as tables embedded within a row of the parent directory, itself a table (represented as File Metadata in the diagram above). The rows within the File Metadata table represent the various file attributes. The file data extent locations are represented by an embedded stream table, which is a table of offset mappings (and, optionally, checksums). This means that the files and directories can be very large without a performance impact, eclipsing the limitations found in NTFS.

As expected, other global structures within the file system such ACLs (Access Control Lists) are represented as tables rooted within the object table.

All disk space allocation is managed by a hierarchical allocator, which represents free space by tables of free space ranges. For scalability, there are three such tables – the large, medium and small allocators. These differ in the granularity of space they manage: for example, a medium allocator manages medium-sized chunks allocated from the large allocator. This makes disk allocation algorithms scale very well, and allows us the benefit of naturally collocating related metadata for better performance. The roots of these allocators as well as that of the object table are reachable from a well-known location on the disk. Some tables have allocators that are private to them, reducing contention and encouraging better allocation locality.

Apart from global system metadata tables, the entries in the object table refer to directories, since files are embedded within directories.

Robust disk update strategy

Updating the disk reliably and efficiently is one of the most important and challenging aspects of a file system design. We spent a lot of time evaluating various approaches. One of the approaches we considered and rejected was to implement a log structured file system. This approach is unsuitable for the type of general-purpose file system required by Windows. NTFS relies on a journal of transactions to ensure consistency on the disk. That approach updates metadata in-place on the disk and uses a journal on the side to keep track of changes that can be rolled back on errors and during recovery from a power loss. One of the benefits of this approach is that it maintains the metadata layout in place, which can be advantageous for read performance. The main disadvantages of a journaling system are that writes can get randomized and, more importantly, the act of updating the disk can corrupt previously written metadata if power is lost at the time of the write, a problem commonly known as torn write.

To maximize reliability and eliminate torn writes, we chose an allocate-on-write approach that never updates metadata in-place, but rather writes it to a different location in an atomic fashion. In some ways this borrows from a very old notion of “shadow paging” that is used to reliably update structures on the disk. Transactions are built on top of this allocate-on-write approach. Since the upper layer of ReFS is derived from NTFS, the new transaction model seamlessly leverages failure recovery logic already present, which has been tested and stabilized over many releases.

ReFS allocates metadata in a way that allows writes to be combined for related parts (for example, stream allocation, file attributes, file names, and directory pages) in fewer, larger I/Os, which is great for both spinning media and flash. At the same time a measure of read contiguity is maintained. The hierarchical allocation scheme is leveraged heavily here.

We perform significant testing where power is withdrawn from the system while the system is under extreme stress, and once the system is back up, all structures are examined for correctness. This testing is the ultimate measure of our success. We have achieved an unprecedented level of robustness in this test for Microsoft file systems. We believe this is industry-leading and fulfills our key design goals.

Resiliency to disk corruptions

As mentioned previously, one of our design goals was to detect and correct corruption. This not only ensures data integrity, but also improves system availability and online operation. Thus, all ReFS metadata is check-summed at the level of a B+ tree page, and the checksum is stored independently from the page itself. This allows us to detect all forms of disk corruption, including lost and misdirected writes and bit rot (degradation of data on the media). In addition, we have added an option where the contents of a file are check-summed as well. When this option, known as “integrity streams,” is enabled, ReFS always writes the file changes to a location different from the original one. This allocate-on-write technique ensures that pre-existing data is not lost due to the new write. The checksum update is done atomically with the data write, so that if power is lost during the write, we always have a consistently verifiable version of the file available whereby corruptions can be detected authoritatively.

We blogged about Storage Spaces a couple of weeks ago. We designed ReFS and Storage Spaces to complement each other, as two components of a complete storage system. We are making Storage Spaces available for NTFS (and client PCs) because there is great utility in that; the architectural layering supports this client-side approach while we adapt ReFS for usage on clients so that ultimately you’ll be able to use ReFS across both clients and servers.

In addition to improved performance, Storage Spaces protects data from partial and complete disk failures by maintaining copies on multiple disks. On read failures, Storage Spaces is able to read alternate copies, and on write failures (as well as complete media loss on read/write) it is able to reallocate data transparently. Many failures don’t involve media failure, but happen due to data corruptions, or lost and misdirected writes.

These are exactly the failures that ReFS can detect using checksums. Once ReFS detects such a failure, it interfaces with Storage Spaces to read all available copies of data and chooses the correct one based on checksum validation. It then tells Storage Spaces to fix the bad copies based on the good copies. All of this happens transparently from the point of view of the application. If ReFS is not running on top of a mirrored Storage Space, then it has no means to automatically repair the corruption. In that case it will simply log an event indicating that corruption was detected and fail the read if it is for file data. I’ll talk more about the impact of this on metadata later.

Checksums (64-bit) are always turned on for ReFS metadata, and assuming that the volume is hosted on a mirrored Storage Space, automatic correction is also always turned on. All integrity streams (see below) are protected in the same way. This creates an end-to-end high integrity solution for the customer, where relatively unreliable storage can be made highly reliable.

Integrity streams

Integrity streams protect file content against all forms of data corruption. Although this feature is valuable for many scenarios, it is not appropriate for some. For example, some applications prefer to manage their file storage carefully and rely on a particular file layout on the disk. Since integrity streams reallocate blocks every time file content is changed, the file layout is too unpredictable for these applications. Database systems are excellent examples of this. Such applications also typically maintain their own checksums of file content and are able to verify and correct data by direct interaction with Storage Spaces APIs.

For those cases where a particular file layout is required, we provide mechanisms and APIs to control this setting at various levels of granularity.

At the most basic level, integrity is an attribute of a file (FILE_ATTRIBUTE_INTEGRITY_STREAM). It is also an attribute of a directory. When present in a directory, it is inherited by all files and directories created inside the directory. For convenience, you can use the “format” command to specify this for the root directory of a volume at format time. Setting it on the root ensures that it propagates by default to every file and directory on the volume. For example:

D:\>format /fs:refs /q /i:enable <volume>

D:\>format /fs:refs /q /i:disable <volume>

By default, when the /i switch is not specified, the behavior that the system chooses depends on whether the volume resides on a mirrored space. On a mirrored space, integrity is enabled because we expect the benefits to significantly outweigh the costs. Applications can always override this programmatically for individual files.

Battling “bit rot”

As we described earlier, the combination of ReFS and Storage Spaces provides a high degree of data resiliency in the presence of disk corruptions and storage failures. A form of data loss that is harder to detect and deal with happens due to ­­­“bit rot,” where parts of the disk develop corruptions over time that go largely undetected since those parts are not read frequently. By the time they are read and detected, the alternate copies may have also been corrupted or lost due to other failures.

In order to deal with bit rot, we have added a system task that periodically scrubs all metadata and Integrity Stream data on a ReFS volume residing on a mirrored Storage Space. Scrubbing involves reading all the redundant copies and validating their correctness using the ReFS checksums. If checksums mismatch, bad copies are fixed using good ones.

The file attribute FILE_ATTRIBUTE_NO_SCRUB_DATA indicates that the scrubber should skip the file. This attribute is useful for those applications that maintain their own integrity information, when the application developer wants tighter control over when and how those files are scrubbed.

The Integrity.exe command line tool is a powerful way to manage the integrity and scrubbing policies.

When all else fails…continued volume availability

We expect many customers to use ReFS in conjunction with mirrored Storage Spaces, in which case corruptions will be automatically and transparently fixed. But there are cases, admittedly rare, when even a volume on a mirrored space can get corrupted – for example faulty system memory can corrupt data, which can then find its way to the disk and corrupt all redundant copies. In addition, some customers may not choose to use a mirrored storage space underneath ReFS.

For these cases where the volume gets corrupted, ReFS implements “salvage,” a feature that removes the corrupt data from the namespace on a live volume. The intention behind this feature is to ensure that non-repairable corruption does not adversely affect the availability of good data. If, for example, a single file in a directory were to become corrupt and could not be automatically repaired, ReFS will remove that file from the file system namespace while salvaging the rest of the volume. This operation can typically be completed in under a second.

Normally, the file system cannot open or delete a corrupt file, making it impossible for an administrator to respond. But because ReFS can still salvage the corrupt data, the administrator is able to recover that file from a backup or have the application re-create it without taking the file system offline. This key innovation ensures that we do not need to run an expensive offline disk checking and correcting tool, and allows for very large data volumes to be deployed without risking large offline periods due to corruption.

A clean fit into the Windows storage stack

We knew we had to design for maximum flexibility and compatibility. We designed ReFS to plug into the storage stack just like another file system, to maximize compatibility with the other layers around it. For example, it can seamlessly leverage BitLocker encryption, Access Control Lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks. We expect most file system filters to work seamlessly with ReFS with little or no modification. Our testing bore this out; for example, we were able to validate the functionality of the existing Forefront antivirus solution.

Some filters that depend on the NTFS physical format will need greater modification. We run an extensive compatibility program where we test our file systems with third-party antivirus, backup, and other such software. We are doing the same with ReFS and will work with our key partners to address any incompatibilities that we discover. This is something we have done before and is not unique to ReFS.

An aspect of flexibility worth noting is that although ReFS and Storage Spaces work well together, they are designed to run independently of each other. This provides maximum deployment flexibility for both components without unnecessarily limiting each other. Or said another way, there are reliability and performance tradeoffs that can be made in choosing a complete storage solution, including deploying ReFS with underlying storage from our partners.

With Storage Spaces, a storage pool can be shared by multiple machines and the virtual disks can seamlessly transition between them, providing additional resiliency to failures. Because of the way we have architected the system, ReFS can seamlessly take advantage of this.

Usage

We have tested ReFS using a sophisticated and vast set of tens of thousands of tests that have been developed over two decades for NTFS. These tests simulate and exceed the requirements of the deployments we expect in terms of stress on the system, failures such as power loss, scalability, and performance. Therefore, ReFS is ready to be deployment-tested in a managed environment. Being the first version of a major file system, we do suggest just a bit of caution. We do not characterize ReFS in Windows 8 as a “beta” feature. It will be a production-ready release when Windows 8 comes out of beta, with the caveat that nothing is more important than the reliability of data. So, unlike any other aspect of a system, this is one where a conservative approach to initial deployment and testing is mandatory.

With this in mind, we will implement ReFS in a staged evolution of the feature: first as a storage system for Windows Server, then as storage for clients, and then ultimately as a boot volume. This is the same approach we have used with new file systems in the past.

Initially, our primary test focus will be running ReFS as a file server. We expect customers to benefit from using it as a file server, especially on a mirrored Storage Space. We also plan to work with our storage partners to integrate it with their storage solutions.

Conclusion

Along with Storage Spaces, ReFS forms the foundation of storage on Windows for the next decade or more. We believe this significantly advances our state of the art for storage. Together, Storage Spaces and ReFS have been architected with headroom to innovate further, and we expect that we will see ReFS as the next massively deployed file system.

-- Surendra

FAQ:

Q) Why is it named ReFS?

ReFS stands for Resilient File System. Although it is designed to be better in many dimensions, resiliency stands out as one of its most prominent features.

Q) What are the capacity limits of ReFS?

The table below shows the capacity limits of the on-disk format. Other concerns may determine some practical limits, such as the system configuration (for example, the amount of memory), limits set by various system components, as well as time taken to populate data sets, backup times, etc.

Attribute	Limit based on the on-disk format
Maximum size of a single file	2^64-1 bytes
Maximum size of a single volume	Format supports 2^78 bytes with 16KB cluster size (2^64 * 16 * 2^10). Windows stack addressing allows 2^64 bytes
Maximum number of files in a directory	2^64
Maximum number of directories in a volume	2^64
Maximum file name length	32K unicode characters
Maximum path length	32K
Maximum size of any storage pool	4 PB
Maximum number of storage pools in a system	No limit
Maximum number of spaces in a storage pool	No limit

Q) Can I convert data between NTFS and ReFS?

In Windows 8 there is no way to convert data in place. Data can be copied. This was an intentional design decision given the size of data sets that we see today and how impractical it would be to do this conversion in place, in addition to the likely change in architected approach before and after conversion.

Q) Can I boot from ReFS in Windows Server 8?

No, this is not implemented or supported.

Q) Can ReFS be used on removable media or drives?

No, this is not implemented or supported.

Q) What semantics or features of NTFS are no longer supported on ReFS?

The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas.

Q) What about parity spaces and ReFS?

ReFS is supported on the fault resiliency options provided by Storage Spaces. In Windows Server 8, automatic data correction is implemented for mirrored spaces only.

Q) Is clustering supported?

Failover clustering is supported, whereby individual volumes can failover across machines. In addition, shared storage pools in a cluster are supported.

Q) What about RAID? How do I use ReFS capabilities of striping, mirroring, or other forms of RAID? Does ReFS deliver the read performance needed for video, for example?

ReFS leverages the data redundancy capabilities of Storage Spaces, which include striped mirrors and parity. The read performance of ReFS is expected to be similar to that of NTFS, with which it shares a lot of the relevant code. It will be great at streaming data.

Q) How come ReFS does not have deduplication, second level caching between DRAM & storage, and writable snapshots?

ReFS does not itself offer deduplication. One side effect of its familiar, pluggable, file system architecture is that other deduplication products will be able to plug into ReFS the same way they do with NTFS.

ReFS does not explicitly implement a second-level cache, but customers can use third-party solutions for this.

ReFS and VSS work together to provide snapshots in a manner consistent with NTFS in Windows environments. For now, they don’t support writable snapshots or snapshots larger than 64TB.

Virtualizing storage for scale, resiliency, and efficiency

Thu, 05 Jan 2012 19:15:00 GMT -

In this post, we are going to dive into a feature in the Windows 8 Developer Preview. Storage Spaces are going to dramatically improve how you manage large volumes of storage at home (and work). We’ve all tried the gamut of storage solutions—from JBOD arrays, to RAID boxes, or NAS boxes. Many of us have been using Windows Home Server Drive Extender and have been hoping for an approach architected more closely as part of NTFS and integrated with Windows more directly. In building the Windows 8 storage improvements, we set out to do just that and developed Storage Spaces. Of course, the existing solutions you already use will continue to work fine in Windows 8, but we think you will appreciate this new feature and the flexible architecture. As we talk all about consumer electronics next week, thinking about all the media we all have in photos (especially huge digital negatives) and videos, this feature is sure to come in handy. In this post, Rajeev Nagar, a group program manager on our Storage and File System team, details this new feature.

In previous posts we’ve seen folks jump to try to identify edge cases or debug the designs. We’re trying an FAQ approach at the end of this post to see if we can focus the dialog a bit :-) The FAQ also talks about the numerous opportunities to use PowerShell as a management tool for Storage Spaces. 

--Steven

---

By my own admission, I am a digital packrat. My data collection continues to expand and includes some of my most precious memories, including irreplaceable photos and home videos of my children since their birth. For quite some time now, I have sought a dependable, expandable, and easy to use solution that maximizes utilization of my ever-growing collection of USB drives. Further, I want guarantees that my data will always be protected despite the occasional hardware failure.

Windows 8 provides a new capability called Storage Spaces enabling just that. In a nutshell, Storage Spaces allow:

• Organization of physical disks into storage pools, which can be easily expanded by simply adding disks. These disks can be connected either through USB, SATA (Serial ATA), or SAS (Serial Attached SCSI). A storage pool can be composed of heterogeneous physical disks – different sized physical disks accessible via different storage interconnects.
• Usage of virtual disks (also known as spaces), which behave just like physical disks for all purposes. However, spaces also have powerful new capabilities associated with them such as thin provisioning (more about that later), as well as resiliency to failures of underlying physical media.

Before we start exploring Storage Spaces in more detail, I will digress briefly to give you a little more context: some of us have used (or are still using), the Windows Home Server Drive Extender technology which was deprecated. Storage Spaces is not intended to be a feature-by-feature replacement for that specialized solution, but it does deliver on many of its core requirements. It is also a fundamental enhancement to the Windows storage platform, which starts with NTFS. Storage Spaces delivers on diverse requirements that can span deployments ranging from a single PC in the home, up to a very large-scale enterprise datacenter.

Pools and spaces

The figure below illustrates the concept of a storage pool. As you can see, we have taken a pair of 2TB (note we use byte measurements as you see in marketing) USB disks and “pooled them” (logically speaking) for subsequent usage.

From this storage pool, we are free to create one or multiple spaces. Note that once physical disks have been added to a pool, they are no longer directly usable by the rest of Windows – they have been virtualized, that is, dedicated to the pool in their entirety. And although we call this “virtualized,” the storage and reliability provided is very real. The available storage capacity can be utilized though creation of spaces from this pool. In the illustration below, we have carved out one such space from the “My Home Storage” pool.

This virtual disk is usable just like a regular physical disk – you can partition it, format it, and start copying data to it. You will notice, however, that the space has a couple of interesting properties:

• Its logical capacity is listed as 10TB although the underlying physical disks in the pool have only 4TB of total raw capacity. As a result, you no longer need to worry up-front about the size.
• Resiliency is built in by associating the mirrored attribute, which means that there are at least two copies of all data contained within the space on at least two different physical disks. Because the space is mirrored, it will continue to work even if one of the physical disks within the pool fails.

The magic that allows us to create a 10TB mirrored space on 4TB of total raw capacity is called thin provisioning. Thin provisioning ensures that actual capacity is reserved for the space only when you decide to use it, for example, when you copy some files to the volume on the space. Previously allocated physical capacity can be reclaimed safely whenever files are deleted, or whenever an application decides that such capacity is no longer needed. This reclaimed capacity is subsequently available for usage by either the same space, or by some other space that is carved out from the same pool. We achieve all of this through architected cooperation between the underlying file-system (NTFS) and Storage Spaces.

With thin provisioning, you can augment physical capacity within the pool on an as-needed basis. As you copy more files and approach the limit of available physical capacity within the pool, Storage Spaces will pop up a notification telling you that you need to add more capacity. You can do so very simply by purchasing additional disks and adding them to your existing pool.

As you see in the illustration above, we have expanded the raw capacity of the “My Home Storage” pool by purchasing and adding four 3TB disks – of course, you could just as well connect SATA and/or SAS storage in conjunction with USB-connected physical disks, and, grow your pool capacity that way. Once we have added this physical capacity, we don’t need to do anything more to consume it. We can simply keep copying files or other data to the space within the pool and this space will automatically grow to utilize all available capacity within the containing pool, subject to its maximum logical size of 10TB. If needed, you can certainly also increase the maximum logical size of a space.

You do not need to explicitly inform Storage Spaces which of your USB disks should be used for each of the spaces you have created. Behind the scenes, Storage Spaces optimally manages the capacity of each of the physical disks within the storage pool, for all the spaces carved out from the pool.

Another core (also optional) capability associated with a space is resiliency to failure of the physical disks comprising the storage pool. For example, the space we’ve illustrated above is a mirrored space (in other words, it has the mirrored resiliency attribute associated with it). This mirrored setting ensures that we always store at least two (and optionally three) complete copies of data on different physical disks within the pool. This way, despite partial or complete disk failure, you’ll never need to worry about loss of data. As a matter of fact, the physical disks comprising the pool are typically not even visible to other components within Windows or to applications running on your PC. By extension, the fact that some physical disks within the pool have failed, is completely shielded from other Windows components or applications. They continue to operate on the space, completely oblivious to the fact that Storage Spaces is working quietly in the background to maintain data availability. Additionally, upon disk failure, Storage Spaces automatically regenerates data copies for all affected spaces as long as sufficient alternate physical disks are available within the pool.

Resiliency through mirroring

It might be interesting to more closely examine how your data is mirrored on different disks. The illustration below shows how a (two-copy) mirrored space is constructed from a two-disk pool:

In this case, Storage Spaces has allocated physical capacity for the mirrored space in what we call “slabs”, which are multiples of 256MB. Also, for this particular example, half of each slab is mirrored on 2 separate disks. Even if one of the two disks fails, Storage Spaces can continue to deliver your data because at least one copy exists on a non-failed physical disk. When multiple disks are available, Storage Spaces spreads slabs across suitable disks as shown in the six-disk pool below:

When a pool disk fails, Storage Spaces identifies the impacted slabs for all spaces utilizing the failed disk, and reallocates them to any available hot-spare disk or to any other suitable disk within the pool (hot-spares are reserved disks within the pool, only to be used as automatic replacements for failed disks). This self-healing is done automatically and transparently so as to minimize the need for manual intervention. We’ve also optimized for speed to prevent data loss from multiple hardware failures at the same time.

Resiliency through parity

There’s another resiliency attribute, called parity, which directs Storage Spaces to store some redundancy information alongside user data contained within the space, thereby enabling automatic data reconstruction in the event of physical disk failure. While conceptually similar to mirroring, parity-based resiliency utilizes capacity more efficiently than mirrored spaces do, but with higher random I/O overhead. Parity spaces are well suited for storing data such as large home videos, which have large capacity requirements, large sequential (predominantly append) write requests, and an infrequent-to-minimal need to update existing content.

Akin to mirrored spaces, slabs for parity spaces are strewn across available disks (with capacity utilized for parity information) as shown below for a parity space contained within a six-disk pool:

When a disk fails, the parity space recovers equally transparently and automatically as does the mirrored space. For parity spaces, Storage Spaces utilizes the parity information to reconstruct affected slabs for all affected spaces, and then automatically reallocates the slab to utilize any available hot-spare disk or any other suitable disk within the pool (just as it does for mirrored spaces)

The illustration below shows two spaces – one with mirrored resiliency and the other with parity resiliency – carved out from the same pool:

Obviously, both spaces above are thinly provisioned and share the same backing pool (physical disks). Slabs for both spaces are intermingled, and optimally spread over all available physical disks, although each space uses different mechanisms to recover from physical disk failure.

You can access spaces contained within a pool, as long as a simple majority of physical disks comprising the pool are healthy and connected to your PC, a concept called quorum. For example, you will need four of the six disks comprising the My Home Storage pool to be healthy and physically connected to the PC in order to access either the Documents or the Multimedia space. Of course, as previously stated, the resiliency attribute associated with the space determines degree of data availability in the presence of physical disk failure – for example, if the Documents space is three-way mirrored and allowed to use all disks within the pool, you can continue accessing data despite the loss of any two disks.

I’ll explain the virtualization capabilities of Storage Spaces by walking you through a common usage scenario. Imagine that you have just purchased a Windows 8 PC and wish to use this machine as a central repository for much of the digital content in your home or small business. A reasonable setup would involve creation of two resilient spaces – one is a mirrored space for your important documents and the like (these are typically modified more often), and the other is a parity space, for your large multi-media content like home videos and family pictures, which you typically update less often, but view more often. By using the appropriate resiliency scheme, you can optimize for both capacity utilization as well as for best performance.

Logically, your storage configuration would look exactly like the illustration provided above, wherein two spaces with different resiliency attributes are carved out from a single pool. Achieving this is quite simple:

1. Connect your physical disks to your PC via USB
2. Create your pool and the two spaces

You can invoke powershell to create the pool and spaces, as well as to complete more advanced tasks. In our example, we have purchased and connected six physical disks to our PC. Below are the simple PowerShell commands to set up our pool and two spaces:

(a) To create a storage pool:

>$pd = Get-PhysicalDisk
>New-StoragePool -PhysicalDisks $pd –StorageSubSystemFriendlyName *Spaces* -FriendlyName “My Home Storage”

(b) To create the two spaces:

>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –ResiliencySettingName Mirror –Size 10TB –Provisioningtype Thin –FriendlyName “Documents”
>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –ResiliencySettingName Parity –Size 50TB –Provisioningtype Thin –FriendlyName “Multimedia”

Note that the above commands will only work on the forthcoming Windows 8 Beta and subsequent releases. A preliminary version of Storage Spaces is available in the Windows 8 Developer Preview, but the above PowerShell commands will not work in that build. If you are curious to try out Storage Spaces on the Developer Preview build, you can use the below alternative commands:

(a) To create a storage pool:

>$pd = Get-PhysicalDisk
>New-StoragePool -PhysicalDisks $pd –FriendlyName “My Home Storage” –StorageSubSystemFriendlyName *Spaces*

(b) To create the two spaces:

>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –StorageAttributesName Mirror –Size 2TB –ProvisioningScheme Sparse –FriendlyName “Documents”
>New-VirtualDisk –StoragePoolFriendlyName “My Home Storage” –StorageAttributesName Parity –Size 1TB –ProvisioningScheme Sparse –FriendlyName “Multimedia”

Also note that, in the Developer Preview, space sizes were limited to < 2TB. That limitation will be removed in the Beta release. Since the availability of the WDP release, we have also activated many additional features within Storage Spaces.

We now get to take a sneak peak at an alternative easy-to-use tool to configure pools and spaces. Beginning with the forthcoming Windows 8 Beta, you can simply go to Control Panel and walk through the sequence below:

(a) To create our pool and a mirrored space, go to Control Panel, click System and Security, and then Storage Spaces.

Click Create a new pool and storage space.

Select the drives you want to add to the new pool.

Select your resiliency mechanism and other options.

Note that you can assign a drive letter and format the resultant volume as part of creating the space.

(b) To add a couple of disks to an existing pool, select the drives you want to add.

(c) To create an additional parity space, click Create a storage space, and then select Parity from the layout options.

(d) In the event you start running out of capacity, expect a notification like this:

Click the notification to see information about the problem and how to fix it.

That’s all you need to do to start using Storage Spaces. Once the spaces have been created, you can utilize them just like any other “disk.” For example, you can turn on BitLocker for the spaces you have created, as shown below.

There is a lot more to say about the many capabilities of Storage Spaces and how other Windows technologies can also leverage these capabilities – we will continue with this discussion in subsequent write-ups.

I hope you find this new capability intriguing and encourage you to play with it. It will all be available to you as part of the Windows 8 Beta release in addition to the features available in the Developer Preview.

- Rajeev

Storage Spaces FAQ

We know that some of you will still have questions about Storage Spaces, so here is an FAQ that we hope will cover most of them. As we get more questions from you in the Comments, we will try to update this FAQ to be more complete.

Q) I use Windows Home Server with Drive Extender. Is there a tool to help me migrate data from the Drive Extender format to Storage Spaces?

No. You will need to create a pool on a Windows 8 PC with a fresh set of disks. Then, you can simply copy data over from your Drive Extender-based volumes to a space within your pool. The functionality delivered through Storage Spaces is more flexible and better integrated with NTFS, so it will generally be more reliable and useful.

Q) Are Storage Spaces some kind of RAID? If it is, what RAID versions do you implement?

Fundamentally, Storage Spaces virtualizes storage in order to be able to deliver a multitude of capabilities in a cost-effective and easy-to-use manner. Storage Spaces delivers resiliency to physical disk (and other similar) failures by maintaining multiple copies of data. To maximize performance, Storage Spaces always stripes data across multiple physical disks. While the RAID concepts of mirroring and striping are used within Storage Spaces, the implementation is optimized for minimized user complexity, maximized flexibility in physical disk utilization and allocation, and fast recovery from physical disk failures. Given these significant differences in objectives and implementation between Storage Spaces and traditional inflexible RAID implementations, the RAID nomenclature is not used by Storage Spaces.

Q) How does the read performance of a space compare to RAID 0 or RAID 10?

For both mirrored and striped spaces, read performance is very competitive with optimized RAID 0 or RAID 10 implementations.

Q) Can I use a RAID enclosure with Storage Spaces for additional reliability and/or performance? Is that a good idea?

We don’t recommend it. Storage Spaces were designed to work with off-the-shelf commodity disks. This feature delivers easy-to-use resiliency to disk failures, and optimizes concurrent usage of all available disks within the pool. Using a RAID enclosure with Storage Spaces adds complexity and a performance penalty that does not provide any improvement in reliability.

Q) Can I boot from a space?

In Windows 8, you cannot boot from a space. As an alternative, you can continue to use dynamic volumes for booting. At release, we will offer guidance on how you can add appropriately partitioned system/boot disks (with dynamic volumes) to a pool.

Q) What is the minimum number of disks I can use to create a pool? What is the maximum?

You can create a pool with only one disk. However, such a pool cannot contain any resilient spaces (i.e. mirrored or parity spaces). It can only contain a simple space which does not provide resiliency to failures. We do test pools comprising multiple hundreds of disks – such as you might see in a datacenter. There is no architectural limit to the number of disks comprising a pool.

Q) How can I know which physical disk a space is on?

Through PowerShell, you can query the set of physical disks backing a particular space. Since all data is striped across all physical disks backing the space, you have this information.

Q) How will I know when a physical disk fails? How do I replace a failed disk?

If the physical disk is contained within an enclosure that supports the SCSI Enclosure Services protocol, we will activate a red LED (if present) next to the failed physical disk. A standard notification will pop up in the desktop. You can also see information about the failure in the Storage Spaces applet in Control Panel. Here is what that looks like:

Through PowerShell, you can also query disk health to determine if a disk has failed.

Once you’ve detected the failed disk, you can physically disconnect it at any time. Replacing a failed physical disk is easy – after removing the failed disk you simply connect the replacement disk to the PC, and then add the disk to the pool either via PowerShell or via Control Panel .

Q) How do I replace a working drive with a bigger one (or just cycle drives)? Does it require a “rebuild”?

As long as you have created mirrored or parity spaces, you can always simply remove a physical disk within the pool, and add a different (perhaps larger) one. Within a short period of time, the impacted spaces will automatically be resynchronized (the Storage Spaces design optimizes this operation to be faster than traditional RAID rebuilds). You can determine whether all spaces are healthy – i.e. data has been resynchronized so as to maintain the designated number of copies –either via Control Panel or via PowerShell commands.

Q) Can I trigger resynchronization myself?

Yes. If you don’t want to wait for automatic resynchronization to start, you can choose the Repair command via PowerShell, which will initiate resynchronization so long as suitable replacement disks and/or spare capacity is available.

Q) What kind of disks will Storage Spaces work with? Are there any special requirements? What about custom enclosures housing these disks?

You can use Storage Spaces with any physical disk that otherwise works with Windows, connected via USB, SATA, or SAS. If the physical disks are connected via some custom enclosure (e.g. in JBOD configurations), Storage Spaces will utilize the SES protocol (if supported by the enclosure), to identify physical slots where the disks are located. When needed, Storage Spaces will also use SES to light up failure LEDs associated with physical disks (assuming that the enclosure has such LEDs). For Storage Spaces to use enclosure capabilities, the enclosure must conform to the Windows logo requirements. Enclosure vendors have been made aware of these requirements and we expect increasing conformance over time.

If your disks are housed within an enclosure, and if Storage Spaces either does not provide you with slot information associated with the physical disks or does not light up LEDs on the enclosure, you can assume that the enclosure does not conform to Windows logo requirements.

Q) Is there a defrag or CHKDSK equivalent for pools?

No. Storage Spaces optimally utilizes all physical disks. In the event that Storage Spaces metadata on a physical disk becomes corrupt (which will be obvious since the disk health will indicate a problem with the physical disk), you can treat the disk just as you would any other failed disk – simply remove it from the pool. If the physical disk is healthy, you can subsequently re-add it to the pool.

Q) How do I know how many mirrors a given file has?

If your file resides within a NTFS volume on a two-way mirrored space, two copies of all your file data will be maintained. If you configure a three-way mirrored space, there will be three copies.

Q) Can I pick which drive to use for mirrors? For example, if I know a particular disk is faster/better/newer?

Yes. In typical deployments, Storage Spaces will automatically select physical disks from the pool to back your spaces. However, if you so desire, you can manually specify a specific set of physical disks within your pool to back a particular space and thereby control allocation. You can do this via PowerShell options at the time you create the space.

Q) Can I change the maximum size of a space? Are there advantages or disadvantages to just making every space 50TB?

You can increase the logical size of a space at any time via Control Panel or PowerShell. Decreasing the logical size is not supported (or needed), given thin provisioning. It makes no difference whether you specify the initial logical size to be a smaller number (say 1TB) and grow it as needed, or set it to a very large number (say 50TB) right from the beginning. The latter may save you time and effort later.

Q) Can I change the slab size to something other than a multiple of 256MB?

No. The slab size is automatically determined by Storage Spaces based on a multitude of factors to deliver an optimal experience in terms of performance and availability.

Q) Does the pre-defined slab size result in sub-optimal utilization of capacity? For example, what if most of my files are very small? What if they’re all large video files?

The slab size is an internal unit of capacity that we use for provisioning across multiple spaces within the same pool. Its value has no bearing on optimal storage of files, regardless of file size.

Q) Can I move a storage pool from one PC to another, once created? For example, if I have a cage with 6 removable drives?

Yes. Just connect the physical disks comprising the pool to the new PC.

Q) Say I have 3 external enclosures and I remove them one at a time. I then plug them into another Windows 8 PC in reverse order.  Will the new PC think I have a broken pool or will it eventually catch up? What if I never plug in one of the enclosures?

You can plug enclosures back in in any order. When Storage Spaces detects a sufficient number of disks for quorum, it activates the pool and contained spaces. You can plug in more enclosures later. If the data on any disks becomes out of sync, Storage Spaces will automatically sync them. Even if you never plug in some enclosures, as long as Storage Spaces detects the minimum number of disks needed, you can continue working with your data. Both via PowerShell and via Control Panel, Storage Spaces informs you that a few physical disks are missing, thereby encouraging you to plug them back in.

Q) You mentioned that quorum for the pool requires a simple majority of healthy and connected physical disks. Does that mean I always need to have an even number of physical disks in the pool? Or do I need an odd number of physical disks? What about two-disk pools?

There is no requirement for an even or odd number of physical disks. Storage Spaces correctly handles two-disk pools and continues delivering resiliency to failures for a two-way mirrored space contained within such a pool, even if one physical disk fails or is disconnected.

Q) What happens when I plug physical disks comprising a pool into a Windows 7 machine?

Windows 7 does not support Storage Spaces and will treat the physical disks just as it would any disk with an unfamiliar partitioning scheme.

Refresh and reset your PC

Wed, 04 Jan 2012 18:00:00 GMT -

The power of personalization is something we all love about PCs, but sometimes there is good reason to want to roll back to an earlier state. Most consumer electronics devices today can be reset to some factory state, and so we built this capability into Windows 8 too. Desmond Lee is a program manager on the Fundamentals team and authored this post about “push-button reset.”
--Steven

---

Many consumer electronic devices these days provide a way for customers to get back to some predefined “good” state. This ranges from the hardware reset button on the back of a wireless network router, to the software reset option on a smartphone. We’ve built two new features in Windows 8 that can help you get your PCs back to a “good state” when they’re not working their best, or back to the “factory state” when you’re about to give them to someone else or decommission them.

Today, there are many different approaches and tools to get a PC back to factory condition. If you buy a PC with Windows preinstalled, it often comes with a manufacturer-provided tool and a hidden partition that can be used for that specific model of PC. You might also use a third-party imaging product, Windows system image backup, or the tried and true method of a clean reinstall from the Windows DVD. While these tools all provide similar functionalities, they don’t provide a consistent experience from one PC or technique to another. If you are the “go to” person for your friends, relatives, or neighbors when they need help with their PCs, you may find that it’s sometimes necessary to just start over and reinstall everything. Without a consistent experience to do this, you might end up spending more time finding the recovery tool for a specific PC than actually fixing the problems, and this gets even worse if you’re helping someone over the phone.

With Windows 8, there are a few key things that we set out to deliver:

• Provide a consistent experience to get the software on any Windows 8 PC back to a good and predictable state.
• Streamline the process so that getting a PC back to a good state with all the things customers care about can be done quickly instead of taking up the whole day.
• Make sure that customers don’t lose their data in the process.
• Provide a fully customizable approach for technical enthusiasts to do things their own way.

As we began planning for Windows 8, we asked ourselves: “Wouldn’t it be great if you could just push a button and everything is fixed?” We really wanted to focus on the concept of “push button”, which translated into a design goal that represents a simple to use, predictable, and fast solution. We also wanted to build on the process many people already use today when they need to start over: back up your data, reinstall Windows and apps, and restore your data. The strength of this approach is that you start over from a truly clean state, but you still get to keep the things you care about. With that as the basis of the solution, our goal was to make the process much more streamlined, less time-consuming, and more accessible to a broad set of customers.

Our solution in Windows 8 consists of two related features:

• Reset your PC – Remove all personal data, apps, and settings from the PC, and reinstall Windows.
• Refresh your PC – Keep all personal data, Metro style apps, and important settings from the PC, and reinstall Windows.

Reset your PC to start over

In some cases, you might just want to remove everything and start from scratch manually. But in other cases, you’re removing your data from a PC because you’re about to recycle or decommission it. For both of these situations, you can easily reset your Windows 8 PC and put the software back into the same condition as it was when you started it for the very first time (such as when you purchased the PC).

Resetting your Windows 8 PC goes like this:

1. The PC boots into the Windows Recovery Environment (Windows RE).
2. Windows RE erases and formats the hard drive partitions on which Windows and personal data reside.
3. Windows RE installs a fresh copy of Windows.
4. The PC restarts into the newly installed copy of Windows.

(Note that the screenshots below reflect changes that we’re making for Beta, some of which are not yet available in Developer Preview)

Resetting your PC

For those of you who worry about data that may still be recoverable after a standard reset, especially on PCs with sensitive personal data, we also will be providing an option in Windows 8 Beta to erase your data more thoroughly, with additional steps that can significantly limit the effectiveness of even sophisticated data recovery attempts. Instead of just formatting the drive, choosing the “Thorough” option will write random patterns to every sector of the drive, overwriting any existing data visible to the operating system. Even if someone removes the drive from your PC, your data will still not be easily recoverable without the use of special equipment that is prohibitively expensive for most people. This approach strikes a good balance between security and performance – a single pass through your hard drive offers more than enough security for typical scenarios such as donation to a local charity, but does not bog you down for hours or days with multi-pass scrubbing operations that might be required for regulatory compliance if you are dealing with highly confidential business and government data.

Choosing how your data should be removed

Refresh your PC to fix problems

Resetting your PC can take you back to square one if you encounter a problem, but that’s clearly a very heavy weight solution, something you’d only do as a last resort. But what if you could get the benefit of a reset – starting over with a fresh Windows install – while still keeping your stuff intact? This is where Refresh comes in handy. Refresh functionality is fundamentally still a reinstall of Windows, just like resetting your PC as described above, but your data, settings, and Metro style apps are preserved. We have a solution to help you with your desktop apps, too, which I’ll talk about a little later.

The coolest part about Refresh is there’s no need to first back up your data to an external hard drive and restore them afterwards.

Refreshing your PC goes like this:

1. The PC boots into Windows RE.
2. Windows RE scans the hard drive for your data, settings, and apps, and puts them aside (on the same drive).
3. Windows RE installs a fresh copy of Windows.
4. Windows RE restores the data, settings, and apps it has set aside into the newly installed copy of Windows.
5. The PC restarts into the newly installed copy of Windows.

Unlike manually reinstalling Windows, you don’t have to go through the Windows Welcome screens again and reconfigure all the initial settings, as your user accounts and those settings are all preserved. You can sign in with the same account and password, and all of your documents and data are preserved in the same locations they were before. To accomplish this, we actually use the same imaging and migration technologies behind Windows Setup. In fact, the underlying setup engine is used to perform both Reset and Refresh, which also benefit from the performance and reliability improvements we added to setup for Windows 8.

Refreshing your PC

Misconfigured settings are sometimes the cause of problems that lead to customers needing to refresh their PCs. To ensure that Refresh is both effective in fixing problems and in making sure customers don’t lose settings that they might have trouble reconfiguring, we’ve thought a great deal about which settings to preserve. In Windows 8 Beta, some of the settings we’ll preserve include:

• Wireless network connections
• Mobile broadband connections
• BitLocker and BitLocker To Go settings
• Drive letter assignments
• Personalization settings such as lock screen background and desktop wallpaper

On the other hand, we deliberately chose not to preserve the following settings, as they can occasionally cause problems if misconfigured:

• File type associations
• Display settings
• Windows Firewall settings

We will continue to enhance and tune both lists over time based on how we see the feature being used in the Developer Preview and Beta.

Restoring your apps

We preserve only Metro style apps when customers refresh their PCs, and require desktop apps that do not come with the PC to be reinstalled manually. We do this for two reasons. First, in many cases there is a single desktop app that is causing the problems that lead to a need to perform this sort of maintenance, but identifying this root cause is not usually possible. And second, we do not want to inadvertently reinstall “bad” apps that were installed unintentionally or that hitched a ride on something good but left no trace of how they were installed.

It is also important to understand that we cannot deterministically replace desktop apps, as there are many installer technologies as well as custom setup and configuration logic, of which Windows has little direct knowledge. That is why we discourage the use of third-party uninstallers or scrubbers. One simple thing to consider is that many setup and installation programs conditionally implement functionality based on the state of the machine at the time of the install (for example default browser, default photo handler, etc.)

You can, however, cleanly install and uninstall all Metro style apps using the .appx package format. If you’re interested in learning more about how Metro style apps work in this regard, check out the following sessions from //build:

• Platform for Metro style apps
• Under the hood: installation and updates for Metro style apps

If you do need to reinstall some desktop apps after you refresh your PC, we save the list of apps that were not preserved in an HTML file, and put this list on the desktop, so you have a quick way to see what you might need to reinstall and where to find them.

One caution is that if any desktop apps you have require a license key, you will need to follow your manufacturer’s instructions for how to reuse the key. This might involve uninstalling the app first, going to a web site, or going through some automated steps by phone, for example.

What if the PC can’t boot?

When your PC is able to boot normally, you can get started with refreshing or resetting it from PC settings. (This is the Metro style app that we called “Control Panel” in the Windows Developer Preview. It is different from the standard Control Panel that you can still use for more complex tasks from the desktop.) The options are easily discoverable, and will be in the same place on every Windows 8 PC. Once launched, you can get through them with just a few clicks, which makes it easy to guide someone through the process over the phone.

However, in some situations, the PC might not boot successfully and you might want to refresh or reset it to get it back to a working condition. In a previous post, Billie Sue Chafins discussed how the boot experience has been redesigned from the ground up, including troubleshooting using Windows RE. Naturally, we’ve made it possible for you to refresh or reset your PC from there as well.

 

Refreshing or resetting your PC from the new boot UI

In Windows 8 Beta, there will also be a tool that you can use to create a bootable USB flash drive, in case even the copy of Windows RE on the hard drive won’t start. You’ll be able to start your PC with the USB drive, and fix problems by refreshing your PC or performing advanced troubleshooting. And if your PC comes with a hidden recovery partition, you’ll even have the option to remove it and reclaim disk space once you’ve created the USB drive.

Refreshing your PC to a state you define, including desktop apps

We know that many of you like to first configure your PC just the way you like it, by installing favorite desktop apps or removing apps that came with the PC, and then create an image of the hard drive before you start using the PC. This way, when you need to start over, you can just restore the image and you won’t have to reinstall the apps from scratch.

With this in mind, we’ve made it possible for you to establish your own baseline image via a command-line tool (recimg.exe). So when you get a Windows 8 PC, you will be able to do the following:

1. Go through the Windows first-run experience to configure basic settings.
2. Install your favorite desktop apps (or uninstall things you don’t want).
3. Configure the machine exactly as you would like it.
4. Use recimg.exe to capture and set your custom image of the system.

After you’ve created the custom image, whenever you refresh your PC, not only will you be able to keep your personal data, settings, and Metro style apps, but you can restore all the desktop apps in your custom image as well. And if you buy a PC that already comes with a recovery image on a hidden partition, you’ll be able to use the tool to switch from using the hidden partition to instead use the custom image you’ve created.

If you’d like to try this out now, a preview version of this tool is included in the Windows 8 Developer Preview. You can try it out by typing the following in a command prompt window running as administrator:

mkdir C:\RefreshImage

recimg -CreateImage C:\RefreshImage

This creates the image under C:\RefreshImage and will register it to be used when you refresh your PC. Again, this is a very early version of the tool, so we know it’s not perfect yet. Rest assured that we’re working hard to get it ready for primetime.

Getting back to productivity quickly

When we started building these features, we knew that ease of use wasn’t going to be enough – refresh and reset had to be fast as well. Many of the recovery tools preloaded with PCs today take an hour or more just to get the PC back to factory condition, and you often still have to spend hours copying back your data and reconfiguring everything. Even solutions that back up and restore the entire hard drive can take a long time, as the time required generally scales with how much data you have.

To give an example of the performance of our solution, we installed a clean copy of Windows on the Developer Preview PC that we gave out to attendees at the BUILD conference, filled most of the drive with data, and measured the time it took to go through various recovery operations:

Recovery operation	Time required
Refreshing the PC	8 minutes 22 seconds
Resetting the PC (quick)	6 minutes 12 seconds
Resetting the PC (thorough, with BitLocker enabled)	6 minutes 21 seconds
Resetting the PC (thorough, without BitLocker)	23 minutes 52 seconds

Compared to a baseline time of 24 minutes 29 seconds for restoring the same contents from a system image backup, most of these times show a considerable improvement.

The beauty of refreshing the PC is that performance isn’t impacted by the amount of data you have. Using the migration technology behind Windows Setup, your data never leaves the drive, and they are not physically moved from one location on the disk to another either, hence minimizing disk reads/writes. Restoring a system image from an external drive using the Windows backup utility, on the other hand, took much longer due to the data in the backup, even with the relatively small 64GB drive on the prototype PC. Thoroughly erasing data did take a bit longer than the other operations, as every sector of the drive had to be overwritten. However, you may also notice that when BitLocker drive encryption was enabled on the drive, this process took much less time. This is due to an optimization we employ so that erasing an encrypted drive would require erasing only the encryption metadata, rendering all the data unrecoverable.

A consistent and easy way to get back to a known good state

Sometimes things can go wrong and you just want to get back to a good state quickly, while other times you might want to remove your data before giving a PC to another family member, employee, or co-worker. With Windows 8, we’ve streamlined these processes and made them more accessible to customers with the new refresh and reset features. Here’s a video showing these features in action:

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

We hope you’ll find these features useful and time-saving when you’re fixing your own PC or helping others with theirs.

-- Desmond Lee

Optimizing picture password security

Mon, 19 Dec 2011 17:00:00 GMT -

We wanted to talk a bit more about the security of picture passwords in a follow up post based on some of your comments.  Jeff Johnson, the Director of Development for the User Experience team, is particularly interested in the math and security of this feature and authored this post on how to optimize the security of the picture password.  Since this is a new form of logging on and concerns over security (especially with mobile devices) as well as new authentication techniques (fragility of facial recognition for example or the challenges we've seen with biometrics) it is no surprise folks took to thinking about potential pitfalls in the approach.  Our goal was to provide a convenient mechanism that was clearly no less secure than text passwords (all that math Jeff provided). Below Jeff talks about why this is a robust solution in general.  Keep in mind in reading this that over the years many "best practices" have been established for typed passwords (policies such as numbers+letters+mixed case, length, inability to recycle passwords, no dictionary words, etc.) as well as important cautions (such as avoiding public internet terminals with potential for overhead cameras or keystroke loggers) -- these types of practices all have analogs in the use of picture password as you can imagine.  Jeff outlines some of these and the logic behind the security of the model.  --Steven

---

A question we’ve been asked several times in one way or another is “I care about keeping my machine secure; what are the best practices for creating the most secure sequence of login gestures?” This leads to an interesting (at least to me, as a math guy) analysis. It involves game theory, but first I’ll distill it down to the following best practices.

• Pick a photo that has at least 10 points of interest. A point of interest is an area that can serve as a landmark for a gesture – a point that you would touch, places you would connect with a line, an area you would circle.
• Use a random mixture of gesture types and sequence. While a line is the gesture that has the most permutations, if you always use 3 lines, that actually makes it easier for an attacker, as they can rule out trying sequences with the other gesture types.
• If you choose to use a tap, a line, and a circle, randomly choose the order of those gestures; this creates 6 times the number of combinations as a predictable order.
• For circle gestures, randomly choose whether you draw it clockwise or counterclockwise. Also consider making the size of the circle bigger or smaller than the “expected” size.
• For line gestures, your instinct may be to always draw from left to right, but it is more secure if you randomly choose the direction with which you connect the two points.
• As with all forms of authentication, when entering your picture password, avoid allowing other people to watch you as you sign in.
• Keep your computer in a secure location where unauthorized people do not have physical access to it.  As with any password entry, be aware of line of sight and potential recording devices that intrude on your screen.
• Be aware that smudges on the screen could potentially identify your gestures. Clean your screen thoroughly on a regular basis. Although this increases the risk if you clean, sign in, and then do nothing, the buildup of oils from repeated use is generally easier for an attacker to see (plus, who likes using an oily device?). Note that buildup is more of an issue for entering numeric PINs, when the device is frequently turned on and off and you enter the sequence dozens of times a day (oils can build up in those locations). Periodically look at your screen at an oblique angle while on the picture password login screen and see if there appears to be a pattern pointing to your gesture sequence. If so, either clean your screen or add a handful of additional smudges in the picture password area (which effectively increases the POIs discussed below)

If you follow these tips, you will substantially increase the security of your computer.

As several comments suggested, we also considered shrinking the size of the image and displaying it at random positions and slight rotations on the screen to minimize any risk from smudges.  We knew from usability feedback that decreasing the size of the image both increased the difficulty of properly entering the gesture and made the login experience feel less immersive; however, if there were a significant improvement to security, we wanted to consider the costs and benefits.  What we discovered was that while shifting the image could reduce the buildup of smudges in specific spots, there were even more prominent “clouds” of taps, lines and circles that were identical relative to each other.  With this information, an attacker could easily figure out the gestures relative to each other.  With that information, it was a simple exercise to move them around the picture until they appeared to coincide with significant elements of the picture.  There wasn’t a noticeable improvement in security and we were able to measure significant degradations to the fast and fluid user experience.  In reality, using smudges is very difficult.  When we took tablets that had been used for a number of days by folks, there were typically too many smudges to even begin to deduce their gesture set.  Even when we were given their login sequence and knew what to look for we had limited success.  We included this analysis because we feel it is important that whenever any innovative new technology is introduced that potential attack vectors are disclosed and the technical community can reach a general consensus of the degree of a threat and its potential mitigations.  Of course we also have confidence that screen technologies will continue to improve and smudges will someday seem quaint.

The analysis

It is also interesting to compute the odds of an attack succeeding in various scenarios. As discussed in the previous blog post, gestures are based on a 100 x 100 grid, giving even the simplest gesture (the tap) a potential of 10,000 values (given proximity matching, this number is effectively reduced to 270). In reality, the number of points of interest (POI) is much lower than that – there are only so many memorable locations in a given photograph.

Although there are other ways to structure an analysis, for the purposes of this discussion we will assume that there are a small number of POIs, and all gestures involve only those points. We assume that taps are directly on a POI, circles only come in two sizes (say, small around the point, and larger around the point) and two directions (clockwise and counterclockwise), and lines always connect two POIs. Because this isn’t strictly true, the number of permutations is actually even greater.

Windows provides additional protection for picture passwords (and PINs) by disabling the login mechanism after 5 incorrect tries (you then have to use your conventional password). With this in mind, it is interesting for a given scenario to frame the relative security in two ways.

First, what are the odds that an attacker with full knowledge of your gesture selection methodology would be able to sign in to your machine before the lockout is triggered (we will refer to this as Odds1). If there are x equally likely gesture sequences, then the odds of guessing it in five tries before lockout are 5 / x .

The second interesting view is assume you were given 100 machines each with a password picked randomly according to the rules of the scenario (we will refer to this as Odds100). What are the odds that an attacker could log in to at least one of those machines? Since these are independent events, the odds of this are:
  .

Base scenario

Let’s assume a horribly insecure scenario: Your “picture” is entirely black with a single white dot in the middle of it. Because there is only one POI, only the tap and circle gesture can be used (there is nowhere to connect a line to). Obviously, if I used only the tap gesture, an attacker would have 100% success as the only valid sequence would be three taps on the white dot. Let’s assume we only use circles and no points. There are 4 possible circles we can randomly choose for each gesture. This gives us a total of 4³ = 64 possible gesture sequences. For this scenario, Odds1 is 7.81% and Odds100 is 99.97%. It’s surprising that for a single machine the odds of a successful sign in with my picture password is less than 8% (my intuition would have guessed a higher number), though you can see it is a virtual certainty that with 100 machines, at least one of them would be compromised. While some users might be comfortable with these odds, most security conscious folks and IT admins who manage a population of machines would find this unacceptable.

Let’s now augment the scenario by saying we will randomly choose for each gesture whether it is a tap or a circle. It is tempting to say that this doubles the complexity of each gesture, but it does not. There are 4 possible circles and 1 possible tap, so there are 5 unique gestures giving a total of 125 sequences.

Let’s say that we choose to implement our new “random” methodology as follows: flip a coin to determine if it’s a tap or a circle. If it’s a circle, we’ll randomly decide which of the four possibilities it will be. While this seems nice and random, it is actually less secure than just using only circles. This is because half the time we will pick a gesture for which there is only one possibility (the tap). An attacker would focus their attack on gestures that featured two or three taps and achieve higher success. An ideal attack strategy (there are others with identical odds) would be to test for 3 taps, and then test for two taps followed by each of the four circle types for the 5 attempts before lockout. Instead of the apparent Odds1 of 4% (an improvement over the previous 7.81%), an attacker would actually achieve Odds1 of 25%, more than three times worse than just using circles. Statistics can be tricky!

Fortunately, there is an easy fix to this scenario. For each gesture, we pick a random number between 1 and 5. If it is a 1, we use a tap. Otherwise we use the value to pick one of the 4 circle possibilities. This does yield an Odds1 of 4% (almost twice as good as the first scenario), but the Odds100 is still an abysmal 98.31%.

A slight improvement

Let’s make just a small improvement to our methodology. This scenario involves a picture with only two POIs (it’s really hard to imagine a real photo this simple, so we can pretend it’s a black canvas with two white dots). This allows us to add the line gesture, but there are only two possibilities for it: drawing from the first dot to the second, or from the second to the first.

Learning from the previous example, we will not randomly pick the gesture type and then the gesture. We will sum up all possible gestures and then pick a random number to map with equal probability onto each possible gesture. There are 2 possible taps, 8 possible circles, and 2 possible lines. The total number of gesture sequences is 12³=1728. This gives us an Odds1 of .29% and Odds100 of 25.2%. It is somewhat remarkable that so simple of a picture with only 2 POIs would have odds this low for a successful attack. Even if you had 100 machines to attempt to break into, you would only succeed getting into at least one machine 1 out of 4 tries.

Ramping it up

Let’s assume there are now 5 POIs in your picture. I can begin to imagine some very simple pictures where this might be the case. We now have 5 possible taps, 20 possible circles, and 20 possible lines. This gives us 45³=91,125 possible sequences. Odds1 is now vanishingly small at 0.0055% and Odds100 is also very low at 0.55%. For many users, these odds are sufficient to protect their data.

To the max

Let’s assume you are very security conscious and choose a picture with 10 POIs. There can be debate as to how many POIs a particular photo contains. However, it doesn’t matter how many POIs are “obvious” as long as you pick 10 points that are identifiable to you to randomly choose gestures with. Actually, if some of the points aren’t obvious (but you can still reliably target them), that is a security plus.

We now have 10 possible taps, 40 possible circles, and 90 possible lines. This is a very robust 140³=2,744,000 sequences. Odds1 is vanishingly small at 0.0002%. In fact, you are more than 50 times more likely to win $10,000 with a $1 ticket in the Washington State Select 4 Lottery than you are to have your machine broken into using a picture with 10 POIs! The Odds100 has dropped to 0.018% and even Odds1000 is only 0.18%.

Social engineering

Social engineering is one of the most significant threats to sign-in security of all types, whether password, PIN, or picture password. Using a randomizer to help construct your sign-in sequence is equally useful for each of these methods.

For the technical enthusiast, it is possible to implement the above schemes with a small amount of programming or the use of Excel. However, it would be useful to have a lower tech way of creating a gesture sequence that a larger audience could employ. Of course, we should not be under any illusions that the number of people who seek out these tools and procedures will be any greater than the number who would voluntarily pick strong text passwords if not required by site admins.

Roll of the dice

As a whimsical exercise, I thought it would be fun to come up with an analog way of generating a random gesture sequence. To do this, I chose to employ a six-sided die (D6 for hard core gamers :-)) to generate a 6-POI gesture sequence. In addition to mapping nicely onto the die, a 6 POI picture has the useful property that the number of possible lines (30) exactly equals the number of taps (6) plus circles (24), so it is easy to bifurcate the gesture type as well.

Repeat the following steps for each of the three gestures:

1. Roll the die.
   The number indicates which of the six POIs to use for the gesture (for a line it will be the starting POI).
2. Roll the die again.

• If the die is even, the gesture will be a line
  Roll the die again.
  If the number matches the first roll to pick the initial POI, reroll until you get a different number.
  This number is the second point for the line.
• If the die is odd, the gesture will be a tap or circle
  Roll the die again.
  Use the roll value list below to determine the gesture.
  1 - The gesture is a tap
  2 - The gesture is a small clockwise circle
  3 - The gesture is a small counterclockwise circle
  4 - The gesture is a larger clockwise circle
  5 - The gesture is a larger counterclockwise circle
  6 - Reroll

As expected, the complexity provided by 6 POIs is between the numbers for 5 POIs and 10 POIs. Odds1 is 0.0023% and Odds100 is 0.23%.

We hope you enjoy using the new picture password sign-in as much as we have enjoyed creating it!

--Jeff Johnson

Signing in with a picture password

Fri, 16 Dec 2011 18:30:00 GMT -

Picture password is a new way to sign in to Windows 8 that is currently in the Developer Preview. Let’s go behind the scenes and see how secure this is and how it was built. One of the neat things about the availability of a touch screen is that it provides an opportunity to look at a new way to sign in to your PC. While many of us might prefer to remove the friction of getting to a PC by running without a password, for most of us, and in most situations this is not the case or is at least unwise. Providing a fast and fluid mechanism to sign in with touch is super important, and we all know that using alpha passwords on touch-screen phones is cumbersome. This post is authored by Zach Pace, a program manager on our You Centered Experience team, and looks at the implementation and security of picture password in Windows 8. Just as a note, you can also use a mouse with picture password too, just by using some click and/or drag actions.
--Steven

---

The experience of signing in to your PC with touch has traditionally been a cumbersome one. In a world with increasingly strict password requirements—with numbers, symbols, and capitalization—it can take upwards of 30 seconds to enter a long, complex password on a touch keyboard. We have a strong belief that your experience with Windows 8 should be both fast and fluid, and that starts when you sign in.

Other touch experiences in the marketplace have tried to tackle this problem, with the canonical example being a numeric PIN. A PIN is a great solution: Almost everyone has seen or used one before, and a keypad is simple to use with touch. We knew though, that there was room to improve.

A numeric combination often presents a problem for people because the sequences easiest to remember are typically the least secure. Common number sequences—like 1111, or 1234—are troublesome, but PINs that are composed of common well-known personal dates can also be deduced if an attacker has personal knowledge of the person (much of which is not hard to obtain). In such a case, the number being personal to a person can work against its security. We set out to change the paradigm here: we designed a fast and fluid touch sign-in experience that is also personal to you.

A personal sign-in experience

At its core, your picture password is comprised of two complimentary parts. There is a picture from your picture collection and a set of gestures that you draw upon it. Instead of having you pick from a canned set of Microsoft images, you provide the picture, because it increases both the security and the memorability of the password. You get to decide the content of the picture and the portions that are important to you. Plus, you get to see a picture that is important to you just like many people do on their phone lock screen.

At its core, the picture password feature is designed to highlight the parts of an image that are important to you, and it requires a set of gestures that allow you to accomplish this quickly and confidently. In order to determine the best set of gestures to use, we distributed a set of pictures to a set of study participants and asked them to highlight the parts of the image that were important to them. That’s it, no additional instructions. What we found were people doing three basic things: indicating location, connecting areas or highlighting paths, and enclosing areas. We mapped these ideas to tap, line, and circle, respectively. It’s the minimal set of gestures we found that allowed people to signify the parts of the image most important to them.

There’s also an attribute inherent to circle and line gestures that adds an additional layer of personalization and security: directionality. When you draw either a circle or a line on your selected picture, Windows remembers how you drew it. So, someone trying to reproduce your picture password needs to not only know the parts of the image you highlighted and the order you did it in, but also the direction and start and end points of the circles and lines that you drew.

We also researched using freeform gestures. When we explored the concept, both with design iterations and research, we found the major pitfall of such a system: the time it takes to sign in. As I mentioned above, we wanted a solution that was faster than a touch keyboard. Throughout the evolutionary process of this feature we used the time taken to sign in using a touch keyboard as a benchmark to judge the success of our methods. We found that when people were allowed to use freeform gestures, it took them consistently longer to sign in. They were slowed down by the concept, feeling that they needed to be unnecessarily precise and trace fine details in an image.

Because people were highlighting areas instead of fine detail, we found that using a limited set of gestures was on average more than three times as fast as the freeform method. We also found that with repeated use, people using the gesture set were consistently able to complete the task in under four seconds, compared to an average of 17 seconds for the freeform model. After continued use of the freeform method, we found many participants asked to change their freeform gestures, picking simple lines and locations instead.

How it works

Once you have selected an image, we divide the image into a grid. The longest dimension of the image is divided into 100 segments. The shorter dimension is then divided on that scale to create the grid upon which you draw gestures.

To set up your picture password, you then place your gestures on the field we create. Individual points are defined by their coordinate (x,y) position on the grid. For the line, we record the starting and ending coordinates, as well as the order in which they occur. We use the ordering information to determine the direction the line was drawn in. For the circle, we record a center point coordinate, the radius of the circle, and its directionality. For the tap, we record the coordinate of the touch point.

When you attempt to sign in with Picture Password we evaluate the gestures you provide, and compare the set to the gestures you used when you set up your picture password. We take a look at the difference between each gesture and decide whether to authenticate you based on the amount of error in the set. If a gesture type is wrong—it should be a circle, but instead it’s a line—authentication will always fail. When the types, ordering, and directionality are all correct, we take a look at how far off each gesture was from the ones we’ve seen before, and decide if it’s close enough to authenticate you.

As an example, let’s take a look at the tap gesture. The tap is the least complex of the three gestures both in number of unique permutations and in the subsequent analysis. When considering whether the spot that you’ve tapped matches a reference spot, our scoring function compares the distance between the gesture you recorded as part of your picture password and the one that you just performed. The score decreases from 100% for a perfect match to 0% when sufficiently far away. Points match when the score is >= 90%. Here is a visual representation of the scoring function for a point in the immediate vicinity of a 100% match:

The area that is scored a match is a circle of radius 3. For any specific tap, a total of 37 (X,Y) locations will return a match. We perform similar calculations for the variables associated with lines and circles.

Security and gesture count

When we took a look at the number of gestures that would be required to use picture password we considered security, memorability, and speed. We sought to balance these often competing attributes to achieve an optimal user experience that would also be secure to use. In order to determine the appropriate gesture count that would meet our security goals, we compared picture password with different authentication methods, namely PIN and plain text password.

The analysis of the number of unique PINs is trivial. A 4-digit PIN (4 digits with 10 independent possibilities each) means there are 10⁴ = 10,000 unique combinations.

When looking at alphanumeric passwords, the analysis can be simplified by assuming passwords are a sequence of characters comprised of lower case letters (26), upper case letters (26), digits (10), and symbols (10). In the most basic case, when a password is comprised strictly of n lower case letters, there are 26ⁿ permutations. When the password can be any length from 1 to n letters, then there are this many permutations:

For instance, an 8-character password has 208 billion possible combinations, which to most people would seem amazingly secure.

Unfortunately, the way most users pick passwords is far from random. Left to their own devices, people use common words and phrases, names of family members, and so on.

In this scenario, let's assume the user composes their password from all but two lower case letters, one upper case letter, and one digit or symbol; however, the upper case letter and digit/symbol can appear in any position of the password. The number of unique passwords is then:

The following table illustrates how the size of the solution space varies with password length and various character set assumptions.

Password length	Unique passwords
1	n/a
2	n/a
3	81,120
4	4,218,240
5	182,790,400
6	7,128,825,600
7	259,489,251,840
8	8,995,627,397,120

When considering picture password, we can conduct a similar analysis for each of the gesture types. The information in the tables below accounts for both unique gesture positions and the leniency of our recognition algorithm.

For the simplest gesture, the tap, the number of unique gesture sets as a function of number of taps is as follows:

# of taps	Unique gestures
1	270
2	23,535
3	2,743,206
4	178,832,265
5	15,344,276,658
6	1,380,314,975,183
7	130,146,054,200,734
8	13,168,374,201,327,200

The circle gesture has more complexity than a tap, but less than a line. In an attempt to quantify the relative security of a circle, we can assume that an attacker knows the radii is guaranteed to be between 6 and 25 (reducing the work to guess a circle gesture), we will further assume that both X and Y coordinates are known to be between 5 and 95. This makes the potential solution space for a hacker to explore to be as follows:

As a function of number of circles, the number of unique gesture sets is as follows:

# of circles	Unique gestures
1	335
2	34,001
3	4,509,567
4	381,311,037
5	44,084,945,533
6	5,968,261,724,338
7	907,853,751,472,886

The most complex gesture of the three is the line. A line is comprised of two points on a normalized 100 x 100 grid, and an ordering of those points. This nominally results in 100 million possible lines; however, lines must be at least 5 units long, so the number of unique lines is actually 99,336,960. Unlike attempts to guess circles where hackers can make simplifying assumptions that significantly reduce the solution space, there are not any similarly obvious reductions for lines. Lines could just as easily go from edge to edge of the screen as they could be very short segments. The number of matches in the case of the line is as follows:

# of lines	Unique gestures
1	1,949
2	846,183
3	412,096,718
4	156,687,051,477
5	70,441,983,603,740

Now that we understand the security of individual gestures, this data can be combined to assess sets containing multiple gestures. This can be done by summing up the unique gestures for all three gesture types for the specific gesture length n and raise it to the n^th power. This results in the table below, which compares picture password to both PIN and alphanumeric password methods.

Length	10-digit PIN	Simple a-z character set password	More complex character set password	Multi-gesture picture password
1	10	26	n/a	2,554
2	100	676	n/a	1,581,773
3	1,000	17,576	81,120	1,155,509,083
4	10,000	456,976	4,218,240	612,157,353,732
5	100,000	11,881,376	182,790,400	398,046,621,309,172
6	1,000,000	308,915,776	7,128,825,600
7	10,000,000	8,031,810,176	259,489,251,840
8	100,000,000	208,827,064,576	8,995,627,397,120

As you can see, the use of three gestures provides a significant number of unique gesture combinations and a similar security promise to a password of 5 or 6 randomly chosen characters. Additionally, using three gestures ensures a Picture Password that is easy to remember and quick to use.

In addition to the number of unique combinations, we’ve increased security of the feature by introducing two safeguards against repeated trial attacks. Similar to the lock out feature on phones using PIN, when you enter your picture password incorrectly 5 times, you are prevented from using the feature again until you sign in with your plain text password. Also, picture password is disabled in remote and network scenarios, preventing network attacks against the feature.

To be clear, picture password is provided as a login mechanism in addition to your text password, not as a replacement for it. You should be sure to have a good hint and use safeguarding mechanisms for your text password, which you can still always use to sign in (the sign-in screen provides a one-click mechanism to switch between all available password entry methods).

Securing against smudges

We’ve also taken some practical considerations to protect you if you use Picture Password. People are often concerned with the smudges left behind on a touch screen and how easy or hard it would be to divine your password based on those markings. Because the order of gestures, their direction and location all matter, it makes the prospect of guessing the correct gesture set based on smudging very difficult even in the completely clean screen case, let alone on a screen that sees regular touch use.

The potential threat here is that smudges left from signing in may yield clues as to the authentication sequence. We can compare three way of logging in—touch keyboard, a four-digit PIN, and picture password—to compare the ease of guessing the sign-in sequence. Let's assume the worst case scenario:

1. The user cleans their screen to an absolutely mirror shine.
2. The user touches exactly the minimum places necessary to authenticate.
3. The user then walks away from their machine without touching it further.
4. The attacker steals the tablet and can with 100% accuracy see every gesture used for authentication.

Obviously, this is rather unlikely, but this scenario allows us to compare and contrast the three forms of authentication and their relative vulnerability to this sort of attack.

A PIN will leave a smudge in a known location for each digit used in the code. If there are n digits in the PIN, and all digits are unique (the hardest to deduce case), there will be n! possible ways of ordering the PIN. For a typical 4-digit PIN, this is 24 different combinations.

For an on-screen keyboard, there are also n! ways of ordering an n-character password. For compliant passwords, a person will typically use the Shift key (or another button) to select alternate character sets. This key press will, of course also be visible to the attacker, but it does not indicate when in the sequence the Shift key was utilized. If we make the simplifying assumption that there is only one shifted key in the password, then there are n!⋅n possible passwords to consider.

Gestures also have n! orderings. For every circle and line used in the gesture set, the number of permutations increases by a factor of two. If all gestures are circles or lines, then the possible set of permutations is the same as a password that uses the Shift key,  .

The following table summarizes the number of permutations for each of these methods for various sequence lengths:

Length	PIN	Password	Password with Shift	Tap-only gestures	Line and circle gestures
1	1	1	1	1	2
2	2	2	4	2	8
3	6	6	18	6	48
4	24	24	96	24	384

Again, this is assuming a completely clean screen with only the gestures visible via smudging. If we consider a scenario where the attacker cannot gain any useful information from smudging—either because the machine is very heavily used (and smudged) or because it is mouse and keyboard only—the chances of guessing the correct sequence becomes even more remote. With our three gestures types, directionality, and the requirement that the sequence be at least three gestures long, the possible number of gesture combinations sits at 1,155,509,083, as discussed above.

The final attack that we considered involves points of interest on an image, or areas that people may commonly choose when presented with an image. Even though the research we did showed this kind of attack to be extremely unreliable—the areas people chose and the kind of gestures they drew upon them correlated very poorly in the lab—we can analyze such an attack by assuming a given picture has points of interest. If the user is free to use any combination of taps, circles, and lines, then the total number of permutations is  , where n  is the length of the picture password. This yields the following number of possible combinations:

Points of interest

Length	5	10	15	20
1	75	200	375	600
2	5,625	40,000	140,625	360,000
3	421,875	8,000,000	52,734,375	216,000,000
4	31,640,625	1,600,000,000	19,775,390,625	129,600,000,000

Assuming the average image has 10 points of interest, and a gesture sequence length of 3, there are 8 million possible combinations, making the prospect of guessing the correct sequence within 5 tries fairly remote.

Although we’re very happy with the robustness of a picture password, we know that there are a variety of businesses for which security is paramount, and anything less than a full password is unacceptable. As such, we’ve implemented group policy that gives a domain administrator the freedom to choose whether picture password can be used. And of course, on your home PC, picture password is optional as well.

When we started the process of designing picture password, we knew that we wanted a sign-in method that was fast, fluid, and personal to each and every user of Windows 8, but still had a robust security promise. Through our research and refinement of both the experience and the concept, we believe we’ve hit on a method of signing in that’s secure but also a lot of fun to use. We love picture password and the additional personal flavor it brings to Windows 8, and we hope you do too!

-- Zach

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

Protecting your digital identity

Wed, 14 Dec 2011 20:00:00 GMT -

We live in a world of usernames, passwords, and PINs when it comes to using our computing devices connected to the Internet. These are very important elements of the digital economy and providing the infrastructure for these in Windows is serious business. This work starts with the most basic step of signing in to Windows, and then includes the technologies used to secure the myriad of accounts you will come to use over time. In this post we take a look at the architectural improvements to Windows that enable even more secure management of your many passwords. Dustin Ingalls, the author of this post, is a group program manager on the security and identity team.

–Steven

---

One of the challenges that we spent a lot of time thinking about while planning Windows 8 was how to help you manage your digital identity in a way that is both convenient and secure. In today’s world, there are a number of very interesting details with respect to digital identities, how they are used, and how they are protected.

Currently, the most common way people verify their digital identity is by using a password. Passwords are used to sign in to your computer, to your bank, to web merchants, and lots of other places. Our research has shown us that the average person using a PC in the United States typically has about 25 online accounts.⁽¹⁾ That’s a lot to keep track of! In fact, the data also shows that the number of unique passwords across those 25 accounts is only about 6. For folks who spend time thinking about security, that’s a worrisome finding as it shows that the average person reuses the same password quite frequently across accounts. Additionally, given that different websites have different password policies (some require alphanumeric with special characters, some disallow special characters, some have minimum password lengths, some don’t, etc.), it’s likely that the number of unique passwords across accounts would be even lower if websites actually had the same password policies.

On the one hand, that’s completely understandable. Remembering a bunch of different passwords is difficult, especially for accounts that we don’t use frequently. On the other hand, password reuse is very useful to hackers…they know that if they can learn your password for one site, it’s highly likely that you use the same password on other sites. Even worse, an attacker can often use your sign-in information to reset the password for other accounts where the password actually is different. For example, if an attacker can somehow gain access to the password for one of your accounts, there’s a strong probability that you use the same password for one of your web email accounts. Given that there are only a handful of major web email providers, finding yours is often pretty easy. Once an attacker gains access to your email, they can go to other common sites (major banks, major online merchants, etc), and use the “lost password” functionality to send a password reset link to the email account that they’ve already taken over.

(As an aside, the Hotmail team has spent a great deal of effort in redesigning the password recovery process for Hotmail. There are many ways that "bad guys" attempt to compromise online accounts (from all providers) and Hotmail is no different. When your account becomes compromised (or you legitimately forget your password), we have in place a number of security steps to make sure that you, and only you, can restore your account. While these might seem inconvenient, consider the relatively small amount of information you provided in order to sign up. That's why we encourage people to add either a secondary email account, or even better, a mobile phone number to their account information. The latter is especially hard to duplicate or hack. If you do find yourself with a compromised Hotmail account, you can reset your password. And for those of you using public terminals or untrusted environments to access Hotmail, we encourage you to use a single-use password sent to you via SMS.)

Clearly, the overall user name/password framework leads to a set of interesting challenges. We all want the web to be frictionless, easy, and safe. Having to remember a whole bunch of complex passwords generally isn’t perceived as frictionless. However, using the same easy-to-remember password across multiple sites isn’t safe. The ideal solution here involves somehow finding a way to make it both easy and safe to use all of your different digital identities.

In thinking through this challenge, there are two basic approaches to making it both easier and safer to manage your digital identity. One approach is to enable Windows to help you manage your passwords. If you could have complex, unique passwords for each website you visit without having to remember them all, that would certainly be easier than having one easy to remember password – at the same time, the complex password would make the business of compromising your identity much more difficult for hackers. Another approach is to use something other than a password to help protect and establish your identity. There have been a number of alternatives to passwords available for many years—technologies such as One Time Passwords (OTP), certificates, smart cards, etc. However, despite some of the superior security properties of these password alternatives, they haven’t exactly caught on for mainstream use—mostly because they’re just not as easy to use as a password.

With Windows 8, we provide support for both the safe storage of username/password combinations, and technology to support alternate authentication; that is, we try to make it easier for you to enhance the security of your passwords, and easier to use newer and stronger techniques for protecting your digital identity.

Shortcomings of passwords

There are a number of different methods that attackers use to try to obtain your password. The most common methods are:

Phishing. Phishing involves tricking a user into revealing their password directly to the attacker. Common forms of phishing include “please reset your account” emails that either ask you to send in your password, or link to a website that looks like a popular website and ask you to enter your password.

Guessing. Given people’s natural preference for easy to remember passwords, attackers can often gain access to an account by simply running through the top 10 or 20 passwords most commonly in use on the Internet. Attackers can also make use of public information (perhaps based on your public social networking profile) to find other easy to guess passwords based on things like your favorite sports team or favorite pet.

Cracking. In certain situations, an attacker can capture a snippet of data (usually the password’s hash value) and use it to derive your password. There are freely downloadable resources on the Internet that enable attackers to derive passwords less than 8 characters in length very quickly.

Keylogging. If an attacker can successfully install a keylogger on a device, they can record each time you hit a key on your keyboard, and therefore easily pick up name/password combinations. This is an especially common attack on public PCs or kiosks. (That's why, for example, using the single use code instead of a password for Hotmail is a good idea in such situations!)

Improving the security and usability of passwords

There are a number of important steps you can take to help protect against all of these types of attacks. One of the most important steps is to keep your PC clean and free of malware (to help against phishing and keylogging). Windows 8 includes a number of substantial features in this area that we’ve already covered in prior blog posts (Secure Boot, SmartScreen and Windows Defender enhancements, etc). However, some attacks (like guessing and cracking) rely only on password strength, so it’s important to use strong, complex passwords that are unique to each account.

Windows 8 simplifies the task of managing unique and complex passwords in two important ways. The first is by providing a way to automatically store and retrieve multiple account names and passwords for all the websites and applications you use, and do so in a protected manner. Internet Explorer 10 uses the credentials that we store to remember names and passwords for websites you visit (if you choose). In addition, anyone building a Metro style app can use a direct API to securely store and retrieve credentials for that app. (It is important to note that IE respects instructions from websites about saving your credentials – some websites specifically request that passwords not be saved.)

Windows 8 allows you to securely store and manage all of your sign-in credentials

The second important investment in this area was covered in an earlier post by Katie Frigon, Signing into Windows 8 with a Windows Live ID. One of the great things you get when you sign in to Windows with your Windows Live ID is the ability to sync the credentials you’ve stored to all of the Windows 8 PCs that you register as your “Trusted PCs.”

When you store credentials in conjunction with signing in to Windows with your Windows Live ID, Windows enables you to set your password for each account to something that is both complex and unique; since Windows 8 will automatically submit the credential on your behalf, you’ll never need to remember it yourself. If you need to see the actual password at some point later, you can view it in the credential manager shown here, from any of your Trusted PCs.

The same principles that keep your credentials safer on websites and applications also apply to how you sign in to your PC. The password you use to protect the account on your PC must be resilient to guessing and cracking. Windows 8 helps with this, helping you to set a very strong password for sign-in, while at the same time enabling a number of “convenience” sign-in methods such as Picture Password and biometrics. This makes it easy to sign in to your PC, without sacrificing security. We will cover Picture Password and other sign-in methods in more detail in a future post.

It is worth reiterating that signing in to your PC with a Windows Live ID, in addition to making sign-in easier, also offers improved sign-in security and gives you a clear path to recovery if you forget your Windows password. With a local password, if you forget your password, you’re in a tough spot – if you didn’t create a password recovery USB stick, you’re stuck rebuilding your machine from scratch. However, if you sign in to your PC with a Windows Live ID, you can reset your password from another PC. If your Windows Live ID password was stolen somehow, you still have the benefit of a number of Windows Live safety features that are designed to detect compromise and limit your account usage until you can successfully prove that you are the rightful owner of your account and recover your account. The account recovery workflow leverages two-factor authentication features (secondary account proofs) that you set up earlier, such as a mobile phone number or secondary email address (if you haven’t already set these up, we’ll ask you for them the first time you use your Windows Live ID with Windows 8). Also, even if your Windows Live ID is in a compromised state, you will still have full access to your PC since Windows will cache your last “known good” sign-in password (encrypted, of course) and allow you to use that to continue to sign in.

Creating an easy to use alternative to passwords

While a complex and unique password can be highly resistant to guessing and cracking, because it is what we refer to as a “shared” or “symmetric” key, it is still always vulnerable to phishing and keylogging. Since the key is shared between you and whatever you are signing in to, if the attacker can somehow gain access to your secret key, the game is up. However, there are alternatives that offer strong protection against these types of attacks.

One alternative is public/private key pairs. Secure Sockets Layer or Transport Layer Security (SSL/TLS) certificates are an example of this – these are the most commonly used methods for protecting network traffic on the Internet today. Public/private key pairs differ from passwords in that they are an “asymmetric” key – the private key and the public key are different, and knowledge of the public key doesn’t enable the attacker to derive the private key. Put very simply, in a public/private key sign-in scheme, when you want to sign in to a service, the service sends you a sign-in request, you sign the request with your private key, and the service then uses your public key to read the signature, proving cryptographically that the sign-in request was signed by whomever holds the corresponding private key. This is referred to as “proof of possession”. So long as you haven’t lost your private key, there is strong cryptographic proof that you are the real account holder signing in to the service. Since the actual private key is never exchanged, both keylogging and phishing no longer work. There are no keystrokes to log; and worst case, if a user is tricked into using their private key to sign an authentication request for a fake website, nothing useful is provided—the bad guys can’t re-use this information to sign in to the legitimate website.

Although this technology is used extensively on the Internet today, it still hasn’t replaced conventional password sign-in. Why not? The main reason is that strong protection of a private key typically requires dedicated hardware (typical examples of this are hardware security modules (HSMs) and smart cards), and historically, use of such hardware hasn’t been very convenient— if you lose the hardware or don’t have it with you, you can’t sign in.

Windows 8 has a number of new features that make it much easier for both users and application developers to make use of public/private key methods. Windows already provides fairly extensive support for use of key pairs and certificates; but strong protection of the private key, as I mentioned earlier, typically relied on HSMs or smart cards. Windows 8 includes a new Key Storage Provider (KSP), which provides easy, convenient use of the Trusted Platform Module (TPM) as a way of strongly protecting private keys. A TPM is a trusted execution environment found on many business-class PCs today (and we expect much broader availability of TPMs when Windows 8 ships), which enables a PC to securely store cryptographic keys. Metro-style apps have APIs that make it easy to automatically enroll and manage keys on your behalf. The Windows Dev Center provides a sample banking app that shows developers how to use this API.

The KSP feature is particularly useful for banking and commerce applications, since it provides very strong resilience against the most common types of identity attacks on the Internet today while leveraging hardware inside your PC to prevent malware from stealing your private key.

For organizations and businesses that already use smart cards, we’ve implemented a new feature that overlays the TPM KSP feature and enables the TPM to function as a “virtual smart card.” This solution is more convenient and economical because you don’t need a physical smart card reader, but deployment is also easier because the virtual smart card functionality works with existing smart card applications and management solutions. The virtual smart card feature can be used in place of existing smart cards with any application or solution that is smart card compatible – no server- or application-side changes are required. Also, Windows 8 continues to support cards compliant with the Personal Identity Verification (PIV) standard or the Generic Identity Device Specification (GIDS) standard. By using these standards, deployment of smart cards is made much easier in Windows 8. All of these options are available for signing in to Windows (on domain-joined PCs), apps, websites – anything that was previously accessible using a physical smart card.  This short video shows this in action after it is set up via policy or logon script by your adminsitrator.

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

In a world that is becoming increasingly dependent on maintaining a secure digital identity, we are very passionate about finding ways to make your digital life safer and more secure, without making it more complex. We’ve spent a great deal of time and focus on this in Windows 8, and we are very much looking forward to hearing your feedback!

- Dustin Ingalls

⁽¹⁾ Source: Dinei Florencio and Cormac Herley, A Large Scale Study of Web Password Habits, Microsoft Research. 2007

Windows Store event and blog

Wed, 07 Dec 2011 01:54:00 GMT -

Today at our Windows Store Preview event in San Francisco, Antoine Leblond revealed the business terms and app policies for anyone developing apps for our new Windows Store. He showed some of the great apps that will be available in the Store at Windows 8 Beta, and demonstrated the design, capabilities, and flexibility of the Store as a platform.

Antoine has also just started a new Windows Store for developers blog, where he will provide info and updates related to the Store for anyone interested in developing apps for Windows 8. The new blog is the place where he will announce and discuss any information about the Store and have an ongoing dialog with app developers about terms, policies, revenue opportunities and platform considerations. It will supplement the conversation we’re having here and B8 will continue to be the primary place for our ongoing conversation about engineering Windows 8.

-- Steven

Enabling large disks and large sectors in Windows 8

Tue, 29 Nov 2011 23:30:00 GMT -

One of the most basic services provided by an OS is the file system, and Windows has one of the most advanced file systems of any operating system used broadly. In Windows 7 we improved things substantially in terms of reliability, management, and robustness (for example, automating completely the antiquated notion of "defrag"). In Windows 8 we build on this work by focusing on scale and capacity. Bryan Matthew, a program manager on the Storage & File System team, authored this post.
--Steven

Our digital collections keep growing at an ever increasing rate – high resolution digital photography, high-definition home movies, and large music collections contribute significantly to this growth. Hard disk vendors have responded to this challenge by delivering very large capacity hard disk drives – a recent IDC market research report estimates that the maximum capacity of a single hard disk drive will increase to 8TB by 2015.

Maximum capacity growth over time for single-disk drives
(Source: IDC Study# 228266, Worldwide Hard Disk Drive 2011–2015 Forecast:
Transformational Times, May 2011)

In this blog entry, I’ll discuss how Windows 8 has evolved in conjunction with offerings from industry partners to enable you to more efficiently and fully utilize these very large capacity drives.

The challenges of very large capacity hard disk drives

To start you out with a little bit of context, we will define “very large capacity” disk drives as sizes > 2.2TB (per disk drive). The current architecture in Windows has some limits that makes these drives somewhat tricky to deal with in some scenarios.

Even as hard disk drive vendors innovated to deliver very large capacity drives, two key challenges required focused attention:

• Ensuring that the entire available capacity is addressable, so as to enable full utilization
• Supporting the hard disk drive vendors in their effort to deliver more efficiently managed physical disks – 4K (large) sector sizes

Let’s discuss both of these in more detail.

Addressing all available capacity

To fully understand the challenges with addressing all available capacity on very large disks, we need to delve into the following concepts:

• The addressing method
• The disk partitioning scheme
• The firmware implementation in the PC – whether BIOS or UEFI

The addressing method

Initially, disks were addressed using the CHS (Cylinder-Head-Sector) method, where you could pinpoint a specific block of data on the disk by specifying which Cylinder, Head, and Sector it was on. I remember in 2001 (when I was still in junior high!) we saw the introduction of a 160GB disk, which marked the limit of the CHS method of addressing (at around 137GB), and systems needed to be redesigned to support larger disks. [Editor’s note: my first hard drive was 5MB, and was the size of a tower PC. --Steven]

The new addressing method was called Logical Block Addressing (LBA) – instead of referring to sectors using discrete geometry, a sector number (logical block address) was used to refer to a specific block of data on the disk. Windows was updated to utilize this new mechanism of addressing available capacity on hard disk drives. With the LBA scheme, each sector has a predefined size (until recently, 512 bytes per sector), and sectors are addressed in monotonically increasing order, beginning with sector 0 and going on to sector n where:

n = (total capacity in bytes)/ (sector size in bytes)

The disk partitioning scheme

While LBA addressing theoretically allows arbitrarily large capacities to be accessed, in practice, the largest value of “n” can be limited by the associated disk partitioning scheme.

The notion of disk partitioning can be traced back to the early 1980s - at the time, system implementers identified the need to divide a disk drive into several partitions (i.e. sub-portions), which could then be individually formatted with a file system, and subsequently used to store data. The Master Boot Record partition table (MBR) scheme was invented at the time, which allowed for up to 32-bits of information to represent the maximum capacity of the disk. Simple math informs us that the largest addressable byte represented via 32 bits is 2³² or 2.2TB. Of course, in the 1980s, this seemed a perfectly legitimate practical limitation to impose, considering that the largest consumer disk available then was a whopping 5MB and cost well over $1500!

As early as in the late 1990s, system implementers recognized the need to enable addressing greater than the 2.2TB limit (among other requirements). A group of companies collaborated to develop a scalable partitioning scheme called the GUID Partition Table (GPT), as part of the Unified Extensible Firmware Interface (UEFI) specification. GPT allows for up to 64-bits of information to store the number that represents the maximum size of a disk, which in turn allows for up to a theoretical maximum of 9.4 ZettaByte (1 ZB = 1,000,000,000,000,000,000,000 bytes).

Beginning with Windows Vista 64-bit, Windows has supported the ability to boot from a GPT partitioned hard disk drive with one key requirement – the system firmware must be UEFI. We've already talked about UEFI, so you know it can be enabled as a new feature of Windows 8 PCs. This leads us to the topic of firmware.

Firmware implementation in the PC – BIOS or UEFI

PC vendors include firmware that is responsible for basic hardware initialization (among other things) before control is handed over to the operating system (Windows). The venerable BIOS (Basic Input Output System) firmware implementations have been around since the PC was invented i.e. circa 1980. Given the very significant evolution in PCs over the decades, the UEFI specification was developed as a replacement for BIOS and implementations have existed since the late 1990s. UEFI was designed from the ground up to work with very large capacity drives by utilizing the GUID partition table, or GPT – although some BIOS implementations have attempted to prolong their own relevance and utility by using workarounds for large capacity drives (e.g. a hybrid MBR-GPT partitioning scheme). These mechanisms can be quite fragile, and can place data at risk. Therefore, Windows has consistently required modern UEFI firmware to be used in conjunction with the GPT scheme for boot disks.

Beginning with Windows 8, multiple new capabilities within Windows will necessitate UEFI. The combination of UEFI firmware + GPT partitioning + LBA allows Windows to fully address very large capacity disks with ease.

Our partners are working hard to deliver Windows 8 based systems that use UEFI to help enable these innovative Windows 8 features and scenarios (e.g. Secure Boot, Encrypted Drive, and Fast Start-up). You can expect that when Windows 8 is released, new systems will support installing Windows 8 to, and booting from, a 3TB or bigger disk. Here’s a preview:

Windows 8 booted from a 3 TB SATA drive with a UEFI system

4KB (large) sector sizes

All hard disk drives include some form of built-in error correction information and logic – this enables hard disk drive vendors to automatically deal with the Signal-to-Noise Ratio (SNR) when reading from the disk platters. As disk capacity increases, bits on the disk get packed closer and closer together; and as they do, the SNR of reading from the disk decreases. To compensate for decreasing SNR, individual sectors on the disk need to store more Error Correction Codes (ECC) to help compensate for errors in reading the sector. Modern disks are now at the point where the current method of storing ECCs is no longer an efficient use of space, – that is, a lot of the space in the current 512-byte sector is being used to store ECC information instead of being available for you to store your data. This, among other things, has led to the introduction of larger sector sizes.

Larger sector sizes – “Advanced Format” media

With a larger sector size, a different scheme can be used to encode the ECC; this is more efficient at correcting for errors, and uses less space overall. This efficiency helps to enable even larger capacities for the future. Hard disk manufacturers agreed to use a sector size of 4KB, which they call “Advanced Format (AF),” and they introduced the first AF drive to the market in late 2009. Since then, hard disk manufacturers have rapidly transitioned their product lines to AF media, with the expectation that all future storage devices will use this format.

Read-Modify-Write

With an AF disk, the layout of data on the media is physically arranged in 4KB blocks. Updates to the media can only occur at that granularity, and so, to enable logical block addressing in smaller units, the disk needs to do some special work. Writes done in units of the physical sector size do not need this special work, so you can think of the physical sector size as the unit of atomicity for the media.

As illustrated below, a 4KB physical sector can be logically addressed with 512-byte logical sectors. In order to write to a single logical sector, the disk cannot simply move the disk head over that section of the physical sector and start writing. Instead, it needs to read the entire 4KB physical sector into a cache, modify the 512-byte logical sector in the cache, and then write the entire 4KB physical sector back to the media (replacing the old block). This is called Read-Modify-Write.

Disks with this emulation layer to support unaligned writes are called 4K with 512-byte emulation, or “512e” for short. Disks without this emulation layer are called “4K Native.”

As a result of Read-Modify-Write, performance can potentially suffer in applications and workloads that issue large amounts of unaligned writes. To provide support for this type of media, Windows needs to ensure that applications can retrieve the physical sector size of the device, and applications (both Windows applications and 3^rd party applications) need to ensure that they align I/O to the reported physical sector size.

Designing for large sector disks

Learning from some issues identified with prior versions of Windows, AF disks have been a key design point for new features and technologies in Windows 8; as a result, Windows 8 is the first OS with full support for both types of AF disks – 512e and 4K Native.

To make this happen, we identified which features and technology areas were most vulnerable to the potential issues described above, and reached out to the teams developing those features to provide guidance and help them test hardware for these scenarios.

Issues we addressed included the following:

• Introduce new and enhance existing API to better enable applications to query for the physical sector size of a disk
• Enhancing large-sector awareness within the NTFS file system, including ensuring appropriate sector padding when performing extending writes (writing to the end of the file)
• Incorporating large-sector awareness in the new VHDx file format used by Hyper-V to fully support both types of AF disks
• Enhancing the Windows boot code to work correctly when booting from 4K native disks

This is just a small cross section of the amount of work done to enable across-the-board support for both types of AF disks in Windows 8. We are also working with other product teams within Microsoft and across the industry (e.g. database application developers) to ensure efficient and correct behavior with AF disks.

In closing

NTFS in Windows 8 fully leverages capabilities delivered by our industry partners to efficiently support very large capacity disks. You can rest assured that your large-capacity storage needs will be well handled beginning with Windows 8 and NTFS!

/Bryan

Improving the setup experience

Mon, 21 Nov 2011 20:00:00 GMT -

Installing Windows is a complex operation that provides an incredibly unique capability—the ability to run a new version of Windows on a vast array of hardware configurations and combinations that were designed with no knowledge of a future Windows, even a version with substantial re-architecture of the Kernel. While most people do not experience the full code path of setup/upgrade (because they buy new PCs and choose to get a new version of Windows that way), even orchestrating the new PC “out of box experience” (OOBE) is a complex technical challenge. Our aim in improving setup is to reduce the time from start to finish so that customers can get to Windows and use the full power of Windows to further customize and ultimately enjoy their new Windows experience. This post was written by Christa St. Pierre on our Setup and Deployment team.
–Steven
(Note, we’re taking a break for the US Holiday)

Setup is something that gets a lot of attention from us in any Windows release. It needs to just work reliably across a huge number of variations of hardware and software. This is true whether you are upgrading your own laptop, or you’re an IT pro who is migrating 10,000 desktops in an enterprise using broad deployment tools. For Windows 7 our main focus was on improving successful install rates, and we did a lot of work to improve reliability and deal with many tough (but relatively rare) cases that had caused problems in setting up earlier versions of Windows. This work gave Windows 7 a more reliable setup experience than in any previous Windows release, as measured by lab testing, customer support incidents, and setup telemetry.

For Windows 8, our goal was to continue to improve reliability while also improving the installation experience and raw performance. Not only did we want it to be rock solid, but also faster and easier to use.

A big challenge

Although millions of people choose to upgrade their existing PCs, most people choose to get a new version of Windows preinstalled on a new PC. In the past that often had to do with increasing system requirements in new Windows releases, and the need to purchase new PCs with more power to run the new version. With Windows 7 however, we made a commitment to work on many more existing PCs by keeping system requirements low and maintaining compatibility. We’ve continued that commitment with Windows 8, so many of you with existing PCs can simply upgrade. Looking just at Windows 7 customers, there are currently more than 450 million PCs that will be able to run Windows 8, but we expect that many systems running Windows Vista and even Windows XP will also be eligible.

Support for these PCs running different Windows versions is a big challenge in terms of testing all possible upgrade paths, languages, service packs, architectures, and editions. When you think about it, it is a rather remarkable achievement that hardware designed for one OS can be supported on an OS that did not exist when the hardware was created, especially considering that connecting hardware to software is a fundamental role played by the OS.

There are always complexities involving hardware support. Sometimes PCs are equipped with peripherals that require updated drivers for Windows 8, and in other cases, for any number of reasons, a PC maker decides that a particular model or configuration is not supported on a new version of Windows. There are also complexities in getting software to work seamlessly upon upgrade, particularly utilities that hook into the lowest levels of Windows such as anti-virus, disk format and defrag, or virtualization. While we have a massive test and ecosystem effort, ultimately the final say on support on a new version of Windows for a PC, peripheral, or software package is determined by the maker of that product. Our commitment to keeping things running and bringing forward software is industry leading and continues with Windows 8. At one recent team meeting, a member of our team showed Windows 8 running Excel version 3.0, which is the 16-bit version of Excel from 1990!

Perceived as “difficult”

During planning for Windows 8, we wanted to hear from customers who chose not to upgrade to Windows 7 even though their PCs would run it. In 2010 we commissioned a study of how people make PC purchase decisions, and talked to customers in three global markets to find out more. While the list of reasons as to why a customer chose not to upgrade varied by market, we have received notable feedback that upgrading the PC was perceived as difficult. So even though many customers wanted to upgrade, the current setup experience might be something that just wasn’t easy enough to make them feel confident in doing so.

Different customer needs

Hearing that some customers think it is too difficult really highlights the fact that we have many different customer needs we need to fulfill with setup. Most customers who buy a Windows upgrade from a retailer just want it to be fast and easy, but a few also want to be able to do some more complex things, such as setting up in a multi-boot configuration. And of course, we also have the IT Pro customers, who need to take full control over configurations, install from network as well as media, and add customizations to the setup image. The advanced user’s needs are a lot like those of the IT Pro, both because they require more fine-tuned control and because it’s hard for us to predict exactly which controls they may want to manipulate. For this reason, we have not created a “super advanced setup” mode, but we encourage people who want to create unattended setup configurations for home or work to use our standalone deployment tools. In Windows 7, we provided a Windows Automated Installation Kit, and in Windows 8 we have enhanced that with additional tools in the Windows Assessment and Deployment Kit, which is available for download to MSDN subscribers.

For this post, I’ll talk mostly about the interactive GUI setup experiences, since that’s where we have the most changes. We sought to maintain very high backwards compatibility with existing unattended installation configurations that IT Pros or advanced users have spent time on for Windows 7, so you can expect those to work consistently for Windows 8 as well, without having to start over. So rest assured that your custom deployments continue to be fully supported as before.

Streamlining the end-to-end experience

Leaving aside automated installations and just looking at the typical GUI scenarios, we still wanted to serve two distinct customer groups in the setup user experience:

• People who want an easy way to upgrade to the new release with an absolute bare minimum of hassle
• People who want to do a clean install, and want more control of setup options, disk layout, and partition configuration

The way we approached these needs was based on the realization that the first group typically runs setup in the UI of their current Windows OS (i.e. they launch it like an app), while the second group typically runs setup from boot media. So, rather than trying to rationalize two fairly different experiences and customer requirements, we chose to maintain two setup user experiences: a streamlined setup that you reach by running an .exe from the DVD or via web delivery, and an advanced setup that runs when you boot off of a DVD or USB key. The streamlined setup is a new experience, optimized for ease-of-use, upgrades, and web delivery via download. Advanced setup is the home of all things familiar to the advanced user, including full support for unattended installation, partition selection, and formatting. Under the covers they share all of the same setup engine components. So both experiences benefitted from our ability to focus on a common codebase for performance and reliability enhancements.

Shifting towards web delivery methods

Before going into the detail on the user experience changes, there’s one big change that is important to call out. In the past, if you wanted to buy an upgrade for Windows, it involved purchasing a boxed product from a retail outlet, taking it home, (sometimes being infuriated while trying to open the box,) and inserting a DVD. However, buying boxed software is quickly becoming the exception rather than the rule, with more and more software being purchased online as broadband penetration increases and large-size media downloads become more common. While we will continue to offer boxed DVDs, we are also making it easier than ever to purchase and install online. This includes starting the setup experience online as well, and having one continuous integrated experience from beginning to end. There is also one big advantage that is a favorite of mine. With our web setup experience, we actually “pre-key” the setup image that is downloaded to a unique user, which means that you don’t have to type in the 25-digit product key when you install!

Streamlining - Reducing repetition and integrating experiences

More than 20 million customers downloaded and ran the Windows 7 Upgrade Advisor during the first six months of availability. Many customers also ran Windows Easy Transfer during this same time period. A reasonable (and often recommended) installation experience for Windows 7 followed a flow like this:

1. Download and install Windows 7 Upgrade Advisor
2. Run Windows Upgrade Advisor
3. Run Windows Easy Transfer to save files and settings
4. Run Windows 7 setup and clean install
5. Run Windows Easy Transfer to restore files and settings

This end-to-end experience included 4 different web and client experiences and required the average customer to walk through 60 screens to complete. The primary reason for the high screen count was the repetition of information. We can visualize it something like this:

 
A common Windows 7 installation experience: Upgrade Advisor, Windows Easy Transfer, and Setup

In Windows 8, rather than having Upgrade Advisor, Setup, and Windows Easy Transfer as separate apps or features, we’ve folded them together into one fast and fluid experience in which we first determine if your PC, apps, and devices will work in the new OS, note which things you want to keep (apps, files and/or settings), and then install the new OS.

We’ve also added the capability for setup to resume automatically after certain actions (such as resolving a blocking compatibility problem), which in the past would have required restarting setup again from the beginning.

Here’s what to expect when you launch the new setup experience from the web :

Determining compatibility
The first thing we do is scan the PC to determine compatibility, resulting in a summary report such as this one:

Windows 8 setup compatibility summary

It provides information on the apps and devices that will work in Windows 8, those that won’t work, and any other system information that is useful to know when determining whether or not to purchase and install Windows 8. A detailed compatibility report is also available if you want to print or save the information, or desire more detail about what to expect once you get to Windows 8, including which apps or devices will require updates.

The compatibility data behind the report covers hundreds of thousands of applications and devices, including retail software, OEM preinstalled software, and peripherals. If an application or device ran on Windows 7, our goals is that it should run on Windows 8 too, but in some cases it may require an update or other support from the OEM or vendor. Some applications also have custom installation logic – installing certain components or settings depending on the OS you’re upgrading from (this is particularly true of system utilities and software that is tightly connected to hardware and peripherals). You may need to uninstall and reinstall these types of apps. (This is also a reason to be careful of 3^rd party “app mover” applications, which claim to move apps from one OS to another, as the end result can be unpredictable or broken.) As a reminder, the best drivers for any system are the ones available directly from the PC manufacturer for embedded hardware and from the device manufacturer for peripherals.

Detailed compatibility report

Downloading Windows 8
Next, an integrated download manager provides time estimates, data validation, the ability to pause, resume, and re-download only parts of the file if something goes wrong. Additionally, because we have already scanned the PC to determine compatibility we know which version of Windows 8 to download – eliminating the need to ask questions such as which language or OS architecture to choose.

Downloading the Windows image from the web

Continuing with installation or creating bootable media
Once the download is complete, you are presented a choice to continue the installation, or install on another partition. The latter option takes you to advanced setup, and allows you to save an ISO or create a bootable USB drive before completing other advanced setup options. (This is the option you’ll need to choose if you want to dual boot, for example.)

Creating bootable media from web-based setup

Choosing what to keep
Next is the upgrade choice. You can choose to keep all, some, or none of your personal data depending on the OS you’re upgrading from, and your personal preferences.

Windows 8 setup options for upgrade and migration

The “Windows settings, personal files, and apps” option is akin to the existing “upgrade” option in Windows 7 and Windows Vista, where an in-place upgrade is performed over the current OS, retaining the apps that were previously installed as well as settings and user files on disk.

The “Just personal files” option is a new functionality, which allows you to get a clean install, but still keep your data without a separate tool such as Windows Easy Transfer.

The upgrade options that you might see in the screen above depend on which version of Windows you are upgrading from. Here’s the list of what you can migrate based on your currently installed version of Windows:

You can transfer these…		When upgrading from…
	Windows 7	Windows Vista	Windows XP
Applications	x
Windows settings	x	x
User accounts and files	x	x	x

Clean install is supported across all versions.

Resolving blocking issues
Often you may need to make changes to your PC before you can continue with the installation. Common requirements include things like uninstalling an application, freeing up disk space, or suspending BitLocker. When encountering this scenario in Windows 7 setup, you would simply see a warning message and then you’d have to exit, take care of the clean-up that was listed, and resume setup again from the beginning. In Windows 8, most items listed in the actionable compatibility report (shown below) include a button to help you directly resolve the blocking issue. For example, if an app needs to be uninstalled, clicking a button in this report automatically launches the uninstaller for that particular app. Once the app is uninstalled the report automatically refreshes, and setup continues without having to start again.

Resolving blocking issues directly from the setup experience

This also works in the case where a reboot is needed. For example, if the blocking app requires a reboot after it is uninstalled, setup will resume from where it left off before the reboot.

The result

The scenario we presented at the beginning that included four different wizards and up to 60 screens in a Windows 7 upgrade can now be accomplished in one end-to-end experience and as few as 11 clicks, an improvement of 82% fewer clicks in Windows 8. The exact number of steps you need to take to complete the installation varies based on your existing OS, migration choices, install method, and number of blocking issues you need to resolve to get the PC ready for installation, but the experience is greatly simplified for everyone. We accomplished all of this with no loss of functionality or customization—we simply streamlined the existing experience.

 
The typical Windows 8 installation experience, with integrated advisor, migration, and setup

Improving upgrade performance

If you had a large number of files on your system, you may have seen that installation times in Windows 7 didn’t scale very well. In fact, as you can see in the diagram below, the more user files there are on a PC being upgraded (regardless of the size of the files) the longer the upgrade takes to complete.

 

Windows 7 time to upgrade in relation to the number of files on a PC

Note: Time in this graph represents time to complete the upgrade once the installation is
initiated, and does not include time to download or read files from media.

The reason for this is that in Windows 7, the upgrade process preserved the customer’s applications in the Program Files folder and their files in the Users folder by moving each file to a transport location (so that the original folders can be deleted to make way for the newer installation), and then moving them back again to complete the installation. With music and photo collections, it’s not unusual to have hundreds of thousands of files, so even relatively fast move operations can really add up.

To address this in Windows 8, we have made several modifications to the upgrade engine to reduce the impact on upgrade times.

Moving whole folders
In the past, each file that was preserved across upgrade was moved individually. In Windows 8, instead of moving things file-by-file, we move entire folders, drastically reducing the number of file operations required. This goes a long way towards shrinking the variation in upgrade times due to the amount of data the customer has on the machine.

At a high level, the logic for whether or not we need to move a given folder is:

• Every file in the folder (and its sub folders) is preserved (there are no exclude rules removing some of the files, for example).
• The entire folder is placed on the target OS unchanged.
• The target destination doesn’t already exist (i.e. we don’t have to merge an existing folder on the destination OS with one from the source OS). There are a few exceptions to this rule however – for example, every folder has a desktop.ini file, but we have logic that allows the source folder to overwrite this file, as in many cases the file is only a cache and can be regenerated.

Simplifying the transport
In Windows 7 the transport (this is the place where we store the files and settings being preserved between the old and new operating systems) was comprised of two folders: “Windows.~q” and “Windows.~tr”. In Windows 8 we have simplified this to just one folder. We have repurposed the “Windows.old” naming convention for consistency with clean install (which creates a “Windows.old” folder containing the previous OS in order to be able to roll back should the installation fail). Merging the transport folders into the single Windows.old folder speeds up the upgrade process, as it removes the need to move files between the ~tr and ~q folders.

Switching to hard links
In upgrades to Windows 7, files were moved between the old OS, the transport, and Windows 7 by using file move operations. In upgrades to Windows 8, we use hard link operations instead. This means we can link to the actual data on disk in the transport location without having to physically move the file, which has a significant performance gain. And if something goes wrong with setup and we have to roll back, we just need to delete the hard links, and the files are completely unaffected on disk.

Removing the down-level gather phase
In Windows 7, the files and settings to be preserved across the upgrade were calculated while the previous OS was still running. The registry values and data collected by our upgrade logic were also gathered while running on the old OS. The content of the files was then gathered offline during the Windows Pre-Installation Environment (Windows PE) phase in order to avoid file-in-use issues.

Most of this work has been removed in Windows 8. The gather rules no longer run during upgrade; instead, we just move the following folders into Windows.old when the PC is offline:

• Windows
• Program files
• Program files (x86)
• Users
• Program data

This means that during the “apply” phase of upgrade (once we are running in Windows 8), everything we need to preserve can be extracted from the Windows.old folder (as we touch no other folders during the upgrade), eliminating the need for a gather phase. Speaking of the Windows.old folder, we have also added a new feature that automatically deletes that folder 4 weeks after a successful install, so you don’t have to worry about removing it. Of course, you can still use the Disk Cleanup tool to remove it immediately if you prefer.

The result

In our labs we compared Windows 7 upgrade times to upgrading to a recent Windows 8 build, and found that the variation in upgrade times based on number of files has been virtually eliminated, as shown in the diagram below.

 
Windows 7 vs. Windows 8 time to upgrade
Note: Time in this graph represents time to complete the upgrade once the installation is initiated
and does not include time to download or read files from media.

Additional optimizations for web delivery

As I mentioned above, Windows 8 setup has been designed for online delivery, in addition to the local delivery from a DVD or USB drive. While downloading Windows has been possible in the past, it was primarily a physical media experience made available for download. In Windows 7 upgrades, for example, two copies were created of the download content on the customer’s drive—the compressed download and the extracted contents—requiring ~5 GB. This could be very problematic on space-constrained systems. Additionally, both the compressed and extracted download contents remained on disk, even after a successful installation.

For Windows 8, in addition to the setup experience improvements for web delivery, we also optimized other aspects. Our goal was to minimize the time it takes for the download to complete, verify the integrity of the bits that are downloaded, minimize disk space requirements, and ensure a resilient download experience for the customer. The two main areas of improvement for Windows 8 are constructing optimized download packages, and making sure that downloading is flexible and resilient.

Constructing optimized download packages
The Windows 7 media layout for x86 consists of 874 files and 200 folders, with a number of redundant files both in the media and compressed within install.wim and boot.wim. To efficiently store and transfer the contents of installation media, we typically use ISO files. For example, an ISO created from the x86 client media is 2.32GB. In order to optimize for download in Windows 8, we take the required subset of files for the specific version of Windows being downloaded. After eliminating duplicates and compressing resources, the single-file size is 2.10GB (as compared to 2.32GB), a savings of 9.5%. After this optimized package is created we compress it using an improved compression algorithm specifically for Windows 8 setup, which provides an additional 28% savings. In this example (using the Windows 7 x86 ISO) the size of the download would be reduced from 2.32GB to 1.51GB.

Downloading is flexible and resilient
The download manager included in Windows 8 setup downloads the optimized package containing the new OS and reconstructs the layout required to run through the install process, without leaving duplicate files on the system. The download manager leverages the Background Intelligent Transfer Service (BITS) as the default transport protocol to transfer files from the Internet to the local machine and provides the ability to pause, resume, and restart. It verifies the bits that are downloaded in 10MB increments. If verification fails for a particular increment, the download manager has the ability to re-request only that specific block of data without having to restart the entire download.

The result

In Windows 8, customers do not have to install a separate download manager, mount the ISO to begin the installation, check the hash of the file for verification post-download, manually clean up unneeded files, or restart a download from the beginning should connectivity be interrupted. Setup takes care of all of these steps automatically, providing a fast, resilient, and easy setup experience. And again, this is true whether you just want to run a quick upgrade on an existing installation, or to create boot media for an advanced setup experience – either with GUI or unattended.

Advanced setup for IT Pros

I said at the beginning that this post wouldn’t go into a lot of detail about our automated installation using the Assessment and Deployment Kit (formerly the WAIK) but I know that many readers of this blog may be interested in learning more about it. So, here are a couple of handy configurations that will make it easy for you to customize a bootable USB drive (that you can create as part of our download experience) and automate your installation. For full details about all of the configurations that are possible, check out the Windows 8 ADK (for MSDN subscribers only). If you aren’t an MSDN subscriber check out the Best Practices for Authoring Answer Files article on TechNet. These best practices still apply for Windows 8 and all of the tools and documents referenced in the article are available in the Windows 7 WAIK or the Windows 8 ADK.

Here’s a video demonstrating an advanced setup from a USB flash drive. Note that this experience is not yet available in the Developer Preview build, but will be there in the final release.

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

Key injection

You may have noticed in the screenshots and video above that none show the familiar “type your product key” experience. In the web setup scenario you won’t see those screens because of key injection from the server, but if you boot from media and choose to do an advanced setup, you’ll have to type it in. With “unattend” settings though, you can do your own key injection, so that you can skip this step. This is handy if you’re reinstalling after changing some system components or replacing a drive.

The specific setting that you will need to configure to do this is the ProductKey setting.

Here’s a sample:

<UserData>
   <ProductKey>
      <Key>12345-12345-12345-12345-12345</Key>
      <WillShowUI>Never</WillShowUI>
   </ProductKey>
</UserData>

Automating install

You can also automate other parts of the experience so that you don’t have to manually click through the screens. Here are a couple of other settings that are useful when automating your install:

You can choose the UI language used for Windows:

<InputLocale>0407:00000407</InputLocale> 
<SystemLocale>de-DE</SystemLocale> 
<UILanguage>de-DE</UILanguage> 
<UserLocale>de-DE</UserLocale>

There are additional settings for every UI choice, so you can script it to the point that the install is essentially hands-off from start to finish.

Dual boot configuration

Unattend is also useful when you want to automatically configure the system for booting multiple operating systems. You could do this all manually in the Advanced Setup GUI and BCD configuration, but why do that when you can script it? The unattend framework is very flexible and you can instruct Setup to format, create, or modify partitions on the PC’s disk(s).

Using the DiskID setting you can create and modify partitions. You can then specify the PartitionID setting to install to a specific partition—one that is different from an existing OS partition.

Here’s a sample to install to a specific partition:

<ImageInstall>
    <OSImage>
        <InstallFrom>
            <Path> Z:\sources\install.wim </Path>
        </InstallFrom>
        <InstallTo>
            <DiskID>0</DiskID>
            <PartitionID>1</PartitionID>
        </InstallTo>
        <WillShowUI>OnError</WillShowUI>
        <InstallToAvailablePartition>false</InstallToAvailablePartition>
    </OSImage>
</ImageInstall>

Where to save your unattend answer file

Once you have your answer file configured to your liking you can copy it to the root of your USB media. (Remember, if you use setup from the web, you can still create a bootable USB drive or save to an ISO.) You can also include the file at the root of the DVD media where you burned the ISO, if you chose that route instead. Even cooler, the unattend file doesn’t even have to be on the installation media. In fact you can place the unattend file at the root of a USB flash drive, plug in the flash drive before starting setup, and setup will automatically find it and use it.

An improved setup experience

With Windows 8 setup we have greatly improved both speed and ease of use, while still retaining all of the advanced setup functionality that many customers will demand. We have integrated what was once many separate steps for people to perform when preparing and starting their setup into a streamlined user experience, with a fast and reliable setup engine under the hood. Customers who choose to install Windows from an online source will have a greatly improved experience over what we’ve delivered in the past, with smaller and faster downloads, as well as increased resiliency and control. We hope that you will find these improvements to be a great way to start your experience using Windows 8.

--Christa St. Pierre

Minimizing restarts after automatic updating in Windows Update

Mon, 14 Nov 2011 18:00:00 GMT -

Before the Internet, updates such as service packs and "patches" were impossibly hard to come by. You ordered upgrade "media" or maybe bought a magazine with a CD in it. Of course, the Internet changed all that. In fact, when ftp.microsoft.com was first set up, among the first services was the ability to get updates for MS-DOS and Windows. With the introduction of Windows Update, we invested heavily in building not just a software delivery service, but a commitment to delivering high quality updates in a timely manner. It took some time to get to the point where customers trust these automatic updates, and we're proud of how far we've come. Today Windows Update is one of the largest services on the Internet by several measures, and of course we're using Windows 8 development as a chance to improve the experience of product updates too. This post was authored by Farzana Rahman, the group program manager of our Windows Update group.
–Steven

When it comes to Windows Update, one of the most discussed topics is the disruptiveness of restarts in the course of automatic updating. And for good reason—restarts can interrupt you right in the middle of something important.

The obvious question to ask first is why does the installation of updates even require a restart at all? Ideally, we would like all update installations to happen seamlessly in the background without a restart. But, in reality, there are situations where the installer is not able to update files because they are in use. In these cases, we need to restart your machine to complete the installation. The automatic updating experience thus needs to be able to handle cases where restarts are required.

We know this architectural challenge is one that frustrates administrators and end-users alike, but it does represent the state of the art for Windows. It is important to understand that for many updates, even if you could continue running the existing code that is already in memory, it is that very code that is a security vulnerability (for example), so the risk to the security (or reliability) of the machine would remain until you restart your machine. We'll keep working on this one. In the meantime, applications that support the Windows Restart Manager (introduced in Windows Vista) can return you to precisely where you left off after a restart.

In this blog, I want to talk about some of the improvements we are making to the automatic updating experience in Windows 8, which will make restarts a little less annoying.

First, some facts about Windows Update

Windows Update (or WU, as we like to say within the team) currently updates over 350 million PCs running Windows 7 and over 800 million PCs across all the supported Windows platforms. There are actually many more PCs updated by WU indirectly if you account for our Windows Software Update Server, and for those machines (or customers) that do all updates manually for any number of reasons.

Since its genesis over a decade ago, the Windows Update experience has evolved quite a bit to adapt to a changing ecosystem, especially the changing requirements around security. And Windows Update has been quite successful in updating PCs in time to stay ahead of large-scale exploits against Windows.

Since the introduction of automatic updating, we have constantly worked to tighten the time it takes to distribute new updates to everyone who uses WU. The chart below (figure 1) shows us how fast downloads and installations occur on Windows 7, from the time of release of an update. The speed of each download is primarily determined by the internet connectivity of the PC, something that WU has no control over, so it is interesting to see below that the majority of update activity occurs in the first three days after release. This three-day number is a key one that I will come back to when we talk about improvements in Windows 8.

In one week, 90% of users worldwide who need the update have successfully completed installation, including the restart, with the number of installations pretty much flattening out after that.

Figure 1 – Completed download and installation of updates from time of release of update

The balance of how broadly and how quickly we can update has proven beneficial to our users to the point where updating is mainly viewed as a background maintenance task (and justly so!) with nearly 90% of users choosing to update automatically on Windows 7. That’s 90% of the total user base telling us to automatically install updates without showing any notifications, or asking for confirmation.

	Windows 7
Automatically install updates	89.30%
Notify me before install	2.38%
Notify me before download	3.44%
Never check for updates	4.88%

Figure 2 – Usage of various modes of automatic updating

Automatic updating and restarts on Windows 7

The next logical question to ask is what is the install experience for people who have chosen to automatically install updates? Below, data collected anonymously from WU gives an insight into the various modes of installation for those who have chosen to install automatically.

As you see above, there are 3 main categories of automatic update installations. Here is what we learned from analyzing each category.

Install-at-shutdown – The majority of automatic update users (39%) are updating when they shut down their systems. For these users, there is no automatic restart because the system can complete all steps of the installation during shutdown. This is the least disruptive experience for users, and so we do want to “hitch a ride” whenever we can on user-initiated shutdowns instead of inconveniencing users with a separate restart.

Install-at-scheduled-time - For the 30% who are scheduling automatic updates, their installations start at a scheduled time (the default is 3 AM in the time-zone where the PC is located) or the next time the user logs in (if we miss the 3 AM window). WU automatically completes any restarts necessary to finish the installation. To ensure that you get the chance to save any important files and data before the restart, we show you a 15-minute countdown timer before the restart.

A fifteen-minute countdown timer warns you of the restart

Allowing restarts to occur without user interaction has helped us to rapidly update a major portion of the Windows ecosystem with critical updates. On average, within a week of releasing a critical update, 90% of PCs have installed the update (see Figure1). On the other hand, this behavior of automatic restarts has some unintended consequences for the user. Restarts can occur without notice, and might occur monthly or even more often if there is an out-of-band update. This unpredictability can potentially result in loss of user data. Most of our automatic installs and the subsequent restarts happen at 3 AM, when users are not around to save any important work. We have heard a lot of painful stories of users coming back to their PCs in the morning to find that a restart occurred, and that some important data was lost. In other cases, the user doesn’t lose data, but needs to restart a job that they were in the middle of (for example, a long copy job).

Interactive install - We were surprised to see 31% of users interactively installing updates; of these 31%, approximately 20% have selected to automatically install, but they manually intervene anyway. WU provides a pop-up notification telling you when updates are available if you have selected to automatically install. The notifications are clearly capturing people’s attention, so they click on the notification and interactively install the updates. But this is actually reinforcing an unintended behavior. If you signed up to get automatic updates, you really shouldn’t need to bother interactively installing an update every time one is available. Most installs should occur silently in the background, and WU should notify you only for critical actions (for example, a pending restart). This also matches feedback from customers, who tell us they find the constant notifications to be distracting. Their expectation when they choose automatic updating is that updating will occur automatically. This seems to be a case where making sure people are in control of their PC experience actually resulted in too much information, and ultimately the price of being in control was a feeling of a loss of control.

With these lessons learned, we set about defining a better automatic updating and restart experience for Windows 8.

Solving the challenge around updating and restarts

The question for us on the WU team is always “What is the best way to quickly update the PC while not being intrusive to the user?” Turns out, this is a hard question to answer, and there is no one simple answer.

The challenge we faced was to find the balance between updating with speed and giving notice to the user for upcoming restarts. Clearly, updating and securing the PC before vulnerabilities can be exploited is just as important as it ever was. However, we also want to deliver a better experience around handling restarts and avoiding data loss without compromising our goal of timely updating.

To this end, the guiding principles we used to design the experience were

• The automatic updating experience is not intrusive to users but keeps them aware of critical actions
• Minimize restarts and make them more predictable
• Continue to keep the PC and the ecosystem up-to-date and secure in a timely manner

Windows Update and handling restarts on Windows 8

Based on these principles, we made the following improvements to the Windows 8 updating experience.

WU will consolidate all the restarts in a month, synchronizing with the monthly security release. This means that your PC will only restart when security updates are installed and require a restart. With this improvement, it does not matter when updates that require restarts are released in a month, since these restarts will wait till the security release. Since security updates are released in a single batch on the second Tuesday of every month, you are then getting essentially one restart a month. This simplification helps in three ways: it keeps the system secure in a timely manner, reduces restarts, and makes restarts more predictable.

There is one exception to the rule to wait for the monthly security release, and that is in the case of critical security update to fix a worm-like vulnerability (for example, a Blaster worm). In that case, WU will not wait, but will go ahead and download, install, and restart automatically. But this will happen only when the security threat is dire enough.

WU notifies you of any upcoming automatic restart. Let’s assume that WU has already detected, downloaded, and installed security updates, and now requires a restart. Windows Update will notify you of an upcoming automatic restart through a message on the login screen that will persist for 3 days. Because the majority of update activity occurs in the first three days of the release of each update [see Figure1], we wanted to give you 3 days to allow you to restart at your own convenience. You would restart by selecting “Update and shutdown” or “Update and restart” on the login screen, or by going to Windows Update in the Control Panel. You will no longer see any pop-up notifications or dialogs about pending restarts. Instead, the message appears in a more visible and appropriate place (the log-in screen). The use of the login screen has become ubiquitous even in home environments, as more and more machines become portable.

Here is a timeline view of that experience:

1.  A message about the upcoming restart is shown in the login screen for three days or until the PC is restarted (whichever is sooner). This means you now have three days to restart the PC at your convenience. All you need to do is see the login screen once in 3 days to see the message about the upcoming restart and by default the lock screen will appear after 15-minute idle timeout.

2.  In addition to the restart notification on the login screen, the Power options on the lock screen will change to “Update and restart” immediately after the update occurs, and will include “Update and shutdown” on days two and three, to make the message even more apparent to you. This allows you to restart your PC at your own convenience.

3. If after three days, the restart still has not occurred, then WU will automatically restart your PC for you. In this case, the automatic restart will happen either at the end of the three-day grace period, or, to prevent data loss if WU detects that there are critical applications open at the end of the three-day grace period, it will wait to automatically restart the next time you login. I’ll address this behavior in more depth in the next section.

4. After the restart has occurred, the message on the login screen will go away and the power options will revert to the original choices. We know people would like Windows to automatically log in after the restart, but we strongly advise against doing so, given the potential security issues with this configuration.

Delay the automatic restart if there is potential of losing user data. If the PC has hit the three-day deadline and still needs an automatic restart, WU will only automatically restart the machine if there is no chance of losing the user’s data. That means, if you are not at your PC (i.e. it is locked), if you have applications running in the background, or if there is potentially unsaved work, WU delays the automatic restart until the next time you come back to your machine and log in. At log-in, you will be asked to save your work, and you’ll see a warning that the machine will be restarted within 15 minutes.

Ensure minimal interruption to user activity. Having a restart notification or dialog pop up in the middle of an important presentation, a game or a movie is not a pleasant situation, to say the least. When attempting to automatically restart the PC, if you are in presentation mode, playing a game, or watching a movie full-screen, WU detects this state, and delays the automatic restart until the next available opportune moment or the next time you log back in to the PC.

The experience for business users. For PCs in an enterprise setting, if no policy has been set by the IT administrator, the updating experience is exactly the same as it is for home users. However, an IT administrator can set a policy to prevent auto-restart after automatic installs (just as in Windows 7). If they set this policy, there will be no three-day countdown and no automatic restart. Instead, users will see a message on the login screen indicating that the PC needs to be restarted, and the message persists until the restart occurs. This informs users that a restart is required while keeping them in control of when to restart.

The experience for users in “notify mode.” I also want to address the experience for users who have chosen to be notified before downloading or installing updates (5.82% of the WU user base from Figure2). For a user in this “notify mode,” a message will be shown on the login screen. If you choose to be notified before downloading updates, you will see the login screen message saying “Important updates are ready to be installed” when updates are ready to be downloaded. If you choose to be notified before install, you will see the same login screen message after updates are downloaded, but before they are installed. In either case, you won’t see the message about a pending restart on the login screen since installation is not automatic.

Cumulatively, these improvements help us achieve the balance we are striving for with Windows Update - keeping the PC (and PC ecosystem) up-to-date, without an intrusive experience.

What about updating 3rd party applications?

Lastly but not the least, I want to address the feedback from users who would like WU to update their 3rd-party applications. People clearly find the experience with multiple updaters on the system less than optimal (and we agree!) Each application updater gives you a different experience, you have to remember to go visit each updater to install updates, you never know when or how updaters will run and what they might do, and so on. People would like one updater for the entire system. On the other hand, users have also told us that they trust the quality of updates distributed by WU and hence are comfortable with choosing to automatically update their systems. We would not want to do anything that might reduce trust in the system by encouraging people to take on this management task manually and exposing their PCs to potential vulnerabilities for even short times.

Through WU and the “Microsoft Update” option (opt-in) we also offer updates for Microsoft products and for 3rd-party device drivers, with a common set of setup tools for each. All of these updates are carefully screened, and must adhere to the Windows conventions for updates regarding rollback and recovery, and overall system impact. As an example, drivers we publish through Windows Update go through tests run by the Windows Logo Program for Hardware, which after validating the updates, signs them for authentication. And, we are continuously working to improve the validation system, to deliver better and higher quality drivers. The wide variety of delivery mechanisms, installation tools, and overall approaches to updates across the full breadth of applications makes it impossible to push all updates through this mechanism.  As frustrating as this might be, it is also an important part of the ecosystem that we cannot just revisit for the installed base of software.  

However, as we discussed at the //build/ conference, the new Windows Store will provide a one-stop shop for (free and paid-for) Metro style apps, with an integrated update service to help ensure apps are maintained in a consistent manner. Because of the vetting process for these apps and the commitment from developers to deliver value to customers, we’re able to bring you this improvement as well.  We’ll have more on this topic in future posts as soon as the Store becomes available to you for public testing.

Looking forward to your feedback.

Farzana

Building a power-smart general-purpose Windows

Tue, 08 Nov 2011 23:00:00 GMT -

In this post, we look at the broad topic of developing an OS to reduce power consumption. We've seen an ever-increasing emphasis on power management in the OS from two perspectives. First, as Windows 8 comes to market, it is easy to see two-thirds of all PCs shipping as portable devices operating on batteries some or most of the time. And second, in the workplace, there is an increasing demand for desktop machines with a reduced carbon footprint as we look to save energy wherever we can. In all cases, this goes beyond standby/hibernate/resume performance and gets to the heart of this post which is about reducing the overall power consumption of the OS and providing OS support for power-saving capabilities in modern hardware. Pat Stemen, a program manager on our Kernel team, authored this post.
--Steven

Battery life and power consumption continue to be some of the most important topics in the computing industry. We wanted to give you a look at how we think about power management for Windows 8, and how we measure power consumption on a daily basis. We consider power management a core OS capability that is critical on any chip architecture and any PC form factor.

Our goals

We have 3 goals in mind when engineering Windows 8 power management:

• Let the hardware shine. We built Windows 8 such that the power efficiency of the hardware platform shines through, regardless of whether the system is a SoC-based Windows tablet or an SLI-equipped gaming PC. We designed our power management interfaces in a consistent, standardized way across all platforms. This allows our hardware partners and application developers to focus on their unique innovations and experiences instead of the differences in platform hardware and power management.
• Continue to deliver great battery life. Windows 7 delivered a significant reduction in power consumption and increase in energy efficiency, particularly mobile PC battery life. (In fact, you can read how we thought about it in this e7 blog post.) In Windows 8, we want to maintain that same level of efficiency on existing PCs even as we re-imagine the rest of Windows.
• Enable the smartphone power model. One of the coolest things about the System-on-Chip (SoC) platforms you’ve seen us talk about at CES and //BUILD/ is their capability to quickly enter very low-power idle states. We want to leverage that ultra-low idle power to bring the constant connectivity and instant-on features of the smartphone power model to capable Windows 8 PCs.
  

Why it matters

Most of us are probably familiar with one impact of good power management—long battery life on our mobile devices. Mobile computing devices are highly scrutinized for their battery runtime with good reason—we are using them on battery more often than ever before. Delivering consistently long battery life requires a lot of well-executed coordination between the underlying hardware, operating system, and application software. (Battery capacity and its quality over time also have a big influence on runtime.)

In addition to mobile battery life, the next most tangible impacts of good PC power management are reduced energy costs and environmental impact. The benefits of power-managing enterprise desktop PCs and servers in the datacenter are typically positioned as reducing the amount of money required to power and cool those systems as well as the resulting reduction in greenhouse gasses required to produce the energy for them. It’s hard to underestimate the impact here—very small changes done well in Windows can result in very large positive environmental impact because of our scale. In many markets around the world, increasing electricity consumption is putting more demand on every aspect of the workplace to reduce power consumption. PCs are a significant source of potential savings.

Power management is also in a fine balancing act with system performance and responsiveness. As an example, we can easily reduce processor performance to save power, but as we do so, we increase the time required to process a given workload. Doing a great job of balancing power and performance is a key requirement across the Windows user experience.

We are also observing a lot of tech blogs reviewing the latest processors and hardware platforms, not just on GHz and benchmark performance, but also on the power consumed to execute the workload. These measurements combine power and performance into energy efficiency—how much power does it cost to execute some fixed workload. “Performance per watt” is the same thing.

However, the thing that excites us the most is how power management is fundamental to all aspects of PC platform design. Power management directly impacts attributes including thickness, weight, acoustics (fans and their speed), skin temperature, cost, screen size, resolution, RAM quantity, etc. Hardware that is thin, light, always connected, and runs all day on battery is cool. We love being a part of enabling that for the Windows ecosystem.

Power as a resource

We think of power as a critical system resource, just like CPU utilization, hard disk activity, or memory consumption. Because it is a core resource, we track how much of it Windows consumes daily in each daily build of Windows 8. This allows us to quickly spot changes in power consumption and work closely with the development team to find the root-cause of the issue, develop, and then test any changes required. As power consumption is reduced, we get a new baseline to measure against future daily builds.

Raw power consumption (e.g. how many milliwatts) on any given Windows PC is the result of a number of platform factors including the CPU and chipset, the type and amount of RAM, the speed, type, and capacity of the storage drive, screen size, etc.

In order to have a consistent baseline, we identify a set of reference platforms and measure their power throughout the Windows 8 development process. Our reference platforms are representative of the type of machines commonly used by our customers. They include platforms from each of our major silicon chip partners. We use power consumption in Windows 7 and Windows 7 SP1 as our baseline, and compare that to measurements taken from Windows 8 builds throughout the development process.

Total system idle power measured in our lab for one reference platform.
You can see a change that caused a ~1.25W increase, which was fixed in a subsequent build.
Note that some variance between runs and builds is expected.

We measure the power consumption of many software workloads on each of our reference systems. The workloads we measure are similar to 3^rd party battery benchmarks in that they are designed to be representative of core scenarios that we all do with our machines every day. Our core workloads include Idle, Web Browsing, Video Playback, Audio Playback, and Standby.

You might be thinking that spending time on the idle workload isn’t useful—after all, few people boot their machines and just let them sit at the Start page and do nothing else. While that is true, we think of the idle workload as the floor of power consumption for the system— it’s the minimum amount of activity that the system has when active. Reducing power consumption at idle reduces the base power consumption of most other workloads, including video and audio playback. Plus, a lot of workloads have significant idle time in them—from small amounts of time between keystrokes when typing, to the minutes between slide transitions when giving a presentation.

The common way to measure power for a mobile Windows PC is a battery life rundown test, in which the battery is charged to 100% and then the battery is drained to 0% while repeating the workload. This method works, but is prone to error as the battery capacity naturally degrades over multiple charge/discharge cycles. Each rundown test is an extra charge/discharge cycle, and we test every day. Thus our measurements could drift over time due to battery degradation.

We have a power measurement setup in our performance lab that allows us to provide reference platforms with direct DC power and measure the consumption. We mentioned the lab and overall capability in our IE blog post on browser power consumption, but didn’t mention that we have a number of real reference laptops set up to measure power consumption every day. The power supplies and their metering capability are automated with test software so that we can continuously install Windows, measure power across scenarios, and then analyze the results with each new build of Windows 8.

DC power supplies with built-in measurement capability

Reference platform instrumented for DC power supply and measurement

How software influences power consumption

Software can influence power consumption by consuming resources—CPU, disk, memory, etc.–as each of those resources has a power cost associated. Software also influences power consumption through the OS and driver software responsible for managing hardware power states.

Windows 8 features 3 key innovations to improve how software influences power consumption—the Metro style app model, idle hygiene, and a new runtime device power management framework. We will give you a brief overview of how these innovations improve power consumption in this blog post.

The Metro style application model

Most of us have experienced the influence of software on power consumption first-hand. It might be that you have an app on your phone that goes through battery quickly or you’ve heard the fan turn on in your laptop when playing a game or computing a spreadsheet. These are all examples of applications directly consuming CPU, GPU, network time, disk and/or memory.

One of the new power management innovations in Windows 8 isn’t a power management infrastructure feature; it is the Metro style application model itself. The Metro style application model is designed from the beginning to be power-friendly. The power management benefit is that the model makes it easy for developers to ensure their application is running only at the right time—applications in the background are suspended such that they do not consume resources and power when not in use.

Of course, we recognize that background activity is a critical component of apps that are always connected and responsive. The Metro style application model and the underlying WinRT support background activity through a new set of capabilities called background tasks. (See this Introduction to Background Tasks for more details.) Background tasks make it easy to perform background activity in a power-friendly fashion. They also enable developers to continue to deliver responsiveness and “freshness” in their applications, but the mechanisms are different than the existing Win32 model because of the desire for a fast-and-fluid interface and the other key attributes of Metro style apps (see 8 traits of great Metro style apps).

We’ve engineered background tasks and the overall Metro style application model to enable a new level of app responsiveness, while at the same time considering overall system attributes including power and memory consumption.

 
Task Manager showing suspended Metro style apps

Idle hygiene

Software can have dramatic influence on power consumption even without consuming a lot of resources through intermittent idle activity. We refer to improvements to idle activity as idle hygiene.

Most PC platforms feature processor and chipset idle states that allow the hardware platform to stop the clock or completely turn off power to parts of the silicon when they are unused. These idle states are absolutely critical to enabling long battery life, but they require a minimal residency duration—that is, you have to be idle for long enough to make the transition in and out of the idle state worthwhile in terms of power used. This is because some power is consumed on the way into and out of the idle state. Software most effectively uses these idle states when there are as few exits from the idle state as possible, and the duration of the idle state is as long as possible.

We track the idle efficiency of Windows 8 using built-in ETW Tracing, some additions to the Windows Performance Analyzer, and a basic histogram. Below, you can see the difference in idle durations between Windows 7 and Windows 8. When the screen is on, we’ve already moved the bar significantly from a maximum idle duration of 15.6ms in Windows 7 to 35% of our durations longer than 100ms in Windows 8! With the screen off and during Connected Standby, our idle durations are even longer, currently in the tens of seconds.

Runtime device power management

PCs attain their longest battery life when all devices, including the processor, storage, and peripheral devices enter low-power modes. Almost every device in the modern PC has some kind of power management technology, and runtime device power management determines how we use those technologies seamlessly without impact to the user experience. A really good example of runtime device power management is dimming the automatic display after a timeout in Windows 7.

Just to underscore how important device power management is, we have seen many systems where not enabling a single device’s power management features can easily reduce total battery life by up to 25%! (It’s worth noting here that disabling a device in Device Manager is almost equally bad—most devices are initialized by firmware at their highest power modes and require a device driver to get them to a more nominal power consumption.) You can diagnose some device power management problems using the built-in powercfg.exe utility in Windows 7 with the /ENERGY parameter. The output of /ENERGY is an HTML file that gives you a view of which devices and software are potentially running in a power-consuming state. Of course, using the factory image for your PC that came loaded with OEM and vendor-supplied drivers is almost always the best way to ensure the devices in your PC are well-behaved for power management.

Efficient power management of devices is performed by the driver for the device, in conjunction with the Windows kernel power manager and platform firmware. The power manager makes it easy for the drivers of these devices to implement their power management routines and coordinate any power state transitions with other devices on the platform.

For Windows 8, we’ve built a new device power framework that allows all devices to advertise their power management capabilities and integrate them with a special driver called the Power Engine Plug-in or PEP, designed for SoC systems. The PEP is provided by the silicon manufacturer and knows all of the SoC-specific power management requirements. This allows device drivers like our USB host controller or a keyboard driver to be built once, and still deliver optimal power management on all platforms from SoC-based PCs to datacenter servers.

We are hard at work with all of our ecosystem partners to deliver the low-power and long battery life technologies we all want in our Windows 8 PCs.

--Pat Stemen

Updating live tiles without draining your battery

Wed, 02 Nov 2011 20:00:00 GMT -

One thing that is becoming far more commonplace across all of our “screens” is the idea of lightweight notifications. Originally, Windows Gadgets were to offer this type of functionality—the idea is a quick heads up display for some critical information (news, weather, sports scores, or line of business events are a few examples). However, the startup time and model of Gadgets are not compatible with reducing overall power consumption (something that is important in a desktop and a laptop) or working to deliver the full-screen platform for developers. In addition, the Start screen of Windows 8 provides a much larger surface to have more of these notifications as well as a user-in-control interface for managing the updates (including usage of networking resources). In a modern experience where more and more information is available via push and in structured snippets, this provides a unique opportunity for developers and end-users. In this post, Ryan Haveson writes about the development of Metro style live tiles and how the architecture scales to large numbers of tiles while also reducing the overall power consumption and system load.
–Steven Sinofsky

We all know that performance and battery life are critically important for a successful release of Windows, and your comments continue to emphasize these attributes. @KISSmakesmeSMILE summed it up well by writing:

“…try matching or better yet, surpassing … [competitor’s] battery runtime achievements on light/low load use.”

At the same time, we know that all modern environments (from PCs to TVs to phones) have some form of gadget, widget, or plug-in model that enables at-a-glance information consumption. Watching TV news, sports, or weather shows a structured screen of information with many sources coming together in real time. People expect to be able to quickly check their stocks, weather, email count, next appointment, line of business status, or even social networking status in a matter of seconds before getting right back to whatever else they were doing. In many ways, one could argue the PC has some catching up to do in this area compared to other devices. As we set out to design our notifications infrastructure, our challenge was in how to make the PC feel alive with activity and remain extremely efficient with respect to power and bandwidth usage. @AndyCadley’s words express the goal well:

“Treat all your "Metro" apps as if they are always running (but at zero impact to battery/performance)”

The Start screen also makes this efficient from a user model perspective by giving you a full screen heads up display without interfering with you desktop or Metro style apps while you are focused on those. In addition, not only did we want to make it efficient, we wanted to make sure that you could install as many notifying apps as you want, without having to worry about the impact on performance or battery life.

One thing we have noticed as we are using Windows 8 internally is that the ability to use the Start screen as a unified and highly readable heads up display for line-of-business applications has become a productivity enhancer. We are seeing a lot of interest in apps that are primarily about notifications. With the scalability of our new push notifications platform, Windows 8 can deliver this capability with minimal system impact, which is a big improvement over the multitude of mechanisms that exist in Windows today. It is not hard to see a scenario, especially early on, where even the most hardcore desktop-only person will find a lot of value in the Start screen as a centralized and well-presented (and controlled) notification area that is just a keystroke away.

Goals of the notification platform

Allowing hundreds of app tiles to be alive with activity, and simultaneously making sure that we don’t degrade performance makes it seem like we have contradictory goals. After all, “activity,” by definition, consumes resources: getting a notification from the cloud uses the network, and rendering the notification on a tile uses GPU/CPU resources, etc. In order to get the design right, we knew we had to stay focused on the goals we started out with:

• Allow hundreds of live tiles without degrading performance
• Go beyond balloons, badges and text, with beautiful images
• Make it easy for developers so they can just “fire and forget”
• Achieve real-time delivery so delivering “instant messages” is instant

Based on these goals, the first fundamental architectural decision that we made was that the platform would be data-driven, that is, no app code should run in the background to power the Start screen.

If you think about the anatomy of a notification delivery system, it involves several pieces: logic for when to connect, authentication, local caching, rendering, error handling, back-off algorithms, throttling, etc. In addition, the system has to deal with service-side issues such as knowing when you are connected or not, so it can cache undelivered content and handle complex scenarios for retrying. Can you imagine if every single app with a live tile had its own version of all that client/server code? Not only would you have different bugs in each implementation, but you would have duplicates of essentially the same code for each app loaded in memory, with code that is constantly being paged in and out to the disk. This would be really inefficient because it would mean all of your apps would be running all the time to keep the Start screen alive. Even on a machine with lots of memory, system performance would eventually grind to a crawl.

If you read Bill Karagounis’s post on how we reduced the memory footprint in Windows 8, you know that performance degrades as you increase the number of processes, DLLs, services, etc. that are running. If each live tile was running with its own code, we would not have been able to achieve our first goal of allowing hundreds of live tiles without degrading performance.

Our solution was to build a data-driven model. This means that a developer can express their tile using a set of predefined properties and templates, in this case, using an XML schema. The XML tile data is then sent to the Windows Push Notification Service (WNS) via a simple HTTP POST and then we take care of the rest. All the code for connecting, retrying, authentication, caching, rendering, error handling, etc. is done in a uniform and power-efficient way.

Here is an example of one of the many tile templates that developers can use for their Windows 8 apps. This one consists of a text field and a single image, but there are many other templates to choose from.

Figure 1: Example template (TileWideImageAndText)

Here is the corresponding XML code that describes the above tile:

<?xml version="1.0" encoding="utf-8"?>
<tile>
  <visual lang="en-US">
    <binding template="TileWideImageAndText">
      <image id="1" src="http://www.fabrikam.com/kickflip.png"/>
      <text id="1">First ever surfboard kickflip recorded in Santa 
        Cruz</text>
    </binding>  
  </visual>
</tile>

The decision to use a data-driven model allowed us to achieve the first two goals (performance and a high-fidelity experience), but we still had to figure out how to achieve real-time delivery and fire-and-forget-it efficiency.

There are two high-level design patterns with client/server content delivery: polling & push. Polling means that the client checks with the service on a regular basis (for example, every 90 minutes) to see if there is new content. Push means that when there is new content, the service sends the data down to the client directly.

The only way to support instant notifications with a polling model would be to poll on a sufficiently high frequency (like every 5 seconds), so if a new message arrives, you’d see it pretty much instantly. But doing so would kill our performance goals—with a 5-second poll interval, the network radio stack would never be idle, battery life would be horrible, and desktop machines would always be powered up. It would be a little like talking on your cell phone all day long—your phone’s battery wouldn’t last long. On top of that, it would be extremely wasteful to check the server every 5 seconds for content, since most of the time there would be nothing new. Historically, system tray notifications and desktop Gadgets introduced in Vista have been implemented using a polling mechanism. But with any polling mechanism, the interval is still not short enough for today's real-time services that are instant.

Thus, for Windows 8 we architected a push-based service. This was a big decision because it meant we would need to build a platform at a global scale, eventually powering the tiles for hundreds of thousands of apps and over a billion people. But the value was clear: developers would get super-efficient real-time notifications to their customers for free, without having to build or maintain their own persistent connections to the client.

The push notification platform

Let’s take a closer look at the various components of the platform to explain some of the more subtle parts of the design. In the diagram below you see three key entities:

1. Windows Push Notification Service (WNS): This powers live tiles and toast notifications.
2. App service: This is the web service that a Metro style app runs (e.g. from their existing website), which sends toast notifications and tile updates via WNS. Examples of this would be the back-end service for the Weather app that shipped in the Developer Preview, or a back-end service hosting photos for a social networking app.
3. Windows 8 client platform: This represents the actual PC and the sub-components in the OS that form the plumbing for the end-to-end experience.

Figure 2: The push notifications platform

Let’s walk through a typical usage scenario to illustrate how this works. Suppose that the app service is a social networking site that sends a tile update when someone comments on your photo (this could just as easily be a line of business app that updates me when a bug is assigned to me or an expense report needs attention for example). When there is an update, the app service sends a notification to WNS (Step 1 in the above diagram). From there, WNS pushes the notification down to the client (Step 2). When it is time to show the tile update on the Start screen, the OS fetches that image from the app service based on the URL contained in the notification XML (Step 3). Once the notification and the image are downloaded, the app renders the live tile based on the template specified in the XML, and presents it on the Start screen.

As stated earlier, one of our goals was “fire and forget.” So, to ensure that developers did not have to write complex caching and retry mechanisms for when the PC is not connected (e.g. if it is a laptop that is sleeping), we cache one notification per app in the WNS cloud until the next time that PC is online.

As we designed the client platform components, we wanted to make sure that everything was engineered for high performance and low power consumption. One of the key parts of this was separating the notification payload from the image payload. A typical notification XML is less than 1KB of data, but an image can be up to 150KB. Separating these allowed us to save significant network bandwidth for scenarios where there is a lot of duplication of the images. For example, the image for a tile may be a profile picture of a friend, which your PC can download once and cache locally to be reused. Separating the notification from the image also allowed us to be smart about discarding unused notifications before we go through the expense of downloading the image. If my device screen is off and is sitting in my bedroom while I am at work, there is no point in downloading images for tiles that will just be replaced by subsequent updates before the next time I use the device.

The authentication model

Because live tiles and notifications represent a key part of the app experience, it is important that the communications channel is authenticated and secure—all the way from the app service to the tile on your Start screen.  It would be pretty bad if an app or a rogue web service could just update any tile on your machine.  For that reason, we use an anonymous authentication mechanism that uniquely identifies the connection between the PC and WNS. Apps and app services also authenticate when communicating with WNS.  Authenticating both connections to WNS helps to protect against abuse of live tile updates, such as spoofing attacks. The authentication mechanism used by WNS explicitly ties the application and service together in a way that keeps other applications (or nefarious individuals) from sending content to a tile that they do not own. And of course, all communication takes place over a secure channel.

All of this works regardless of whether or not you sign in to Windows using a Windows Live ID. Of course, as Katie Frigon talked about in her post on signing in with a Windows Live ID, Windows 8 is best when you have a connected account as it enables a lot of enhanced experiences such as app cloud storage, roaming Windows and app settings, and single sign-on to multiple apps.  Because the push notification platform uses an anonymous authentication mechanism, even if you do sign in with a Windows Live ID, the developer of the app can’t use the notification pipeline to discover your Windows Live ID, system info, or location.

Building the service to scale

Earlier in this post we mentioned that we had to engineer the platform to support an incredibly large number of users and apps. To give you an idea of this scale, the graph below shows the number of notifications that apps are sending to Windows 8 per day. As of a couple weeks ago, we were already sending almost 90 million tile updates per day, and we are not even at beta yet!

Figure 3: Notifications per day sent to Windows 8 Developer Preview build

The Stocks app is one of the popular test drive apps from the Developer Preview build. The following graph shows the total number of live tiles registered with this app in the first month since release of the Developer Preview build.

Figure 4: Live tiles registered to the Developer Preview Stocks app

When the Developer Preview was released, we started watching traffic coming through the data centers, carefully monitoring our scale-out. Here is a visualization of the actual geographic distribution of notifications in the first few days after the Developer Preview was released at //build. Note that the data represents units per square mile and was fitted to a logarithmic scale to account for a wide range of density values.

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

The design of WNS is based on the Windows Live Messenger service architecture, and in fact, the service part of the notifications platform was built by the same team. There are not many teams in the world with the expertise and knowledge to be able to build a globally scalable service that can ramp up to such large numbers so quickly. Here are a few statistics to give you an idea of the scale of the Windows Live Messenger service today:

• 300M monthly active users
• 630M daily logins
• 10B daily notifications
• Over 40M peak SOC (simultaneous online connections)
• Over 3000 machines routing messages around the world

Transparency into tile resource usage via Task Manager

We were so passionate about the performance aspect of notifications platform that we added metrics in the new Task Manager to allow you to keep track of how much bandwidth the tile platform is consuming for each of your applications. In general, resource usage for tiles should be relatively low. For those of you running the Developer Preview build, go to the app history tab in Task Manager and look at the “Tiles” column to see how much bandwidth each of your live tiles has consumed over the last 30 days.

Figure 5: Resource usage of live tiles shown in Task Manager app history

Summary

In Windows 8, we set out to design a notifications platform that would provide at-a-glance information, without all the performance and battery life concerns that face traditional plugin and gadget-based models. To that end, every design decision we made was viewed through the lens of performance and battery life efficiency. To make it easy for app developers to participate, we built the Windows Push Notifications Service so they could create live tiles without having to write complicated network connectivity code. And because WNS uses standard web technologies, such as HTTP POST, it’s easy for developers to integrate notifications based on their existing web services.

The result is a notifications platform that delivers at-a-glance information while allowing you to install as many apps as you want without worrying about the impact on performance or battery life.

--Ryan Haveson

Using Task Manager with 64+ logical processors

Thu, 27 Oct 2011 19:00:00 GMT -

Ryan Haveson, a group program manager on the User Experience team, wanted to update folks on some progress with Task Manager since the Windows Developer Preview. In this post you'll find the updated Task Manager tools for managing systems with a large number of logical processors. This is scalability well beyond desktop PCs, and is designed for the server and data center. A big part of Windows development is that the OS scales across a wide range of form factors and CPU architectures.

Note on comments.  Please keep comments up to community standards.  Just a reminder that there is no moderation of comments other than automated spam protection.  –Steven Sinofsky

We talked about the new Task Manager in a previous post, and many of you have installed the Developer Preview and seen it for yourself. There was some interest on this topic so we thought we would take a moment to quickly share with you a feature that just showed up in our daily builds that you will be able to see for yourself in the future, in the Beta release.

The pictures below relate to a feature that server admins and people with access to mega-PC setups with lots and lots of logical processors often ask us about. One key thing to note up front is that here we are talking about logical processors, so if you have a system capable of hyper-threading, you will see multiple logical processors for each physical processor.

For those of you who have access to one of these many-processor systems, you know that the task manager CPU charts in Windows 7 have a few limitations:

• Lack of real-time comparisons: When you are looking at a CPU graph for lots and lots of logical processors, it is the anomalies that are interesting. At scale, it is pretty hard to compare moving line graphs of a 60-second window of CPU utilization to understand what is going on.
• Tiny graphs: When you get to the 64+ logical processor range, the graphs get pretty small. If you are trying to figure out which processors are being heavily used, you really have to squint to figure it out. When you get over 256 logical processors, you can barely read the charts at all.
• Finding the processor ID: If you do identify an anomalous graph, there is no easy way to get the corresponding processor ID.

Below is the Windows 7 Task Manager CPU performance tab on a system with 160 logical processors.

Figure 1: Old Task Manager showing 160 logical processors

As you can see, it is really hard to compare the cells in the CPU Usage History table to each other. The graphs are hard to read, and if you want to compare instantaneous CPU utilization, it is nearly impossible because each cell is showing a moving 60-second graph. Moreover, all the graphs in the CPU Usage History table look identical, so you can’t easily find the processor ID for a specific graph. In our previous post on Task Manager, we discussed the benefits of using a heat map as a visualization to convey and compare large amounts of numerical data. When we looked at designing the graphs for the “many-core view” of the new performance dashboard, a heat map was a natural fit.

In the screen shots below, taken from a current build of Windows 8, it is now easy to see all the logical processors at a glance and know which are being utilized to high and low capacity.

(Note: The screen shots below show Task Manager on a system with 160 logical processors with a simulated workload.)

Figure 2: New Task Manager showing 160 logical processors

In the new CPU graph, you can also get the logical processor ID that maps to each entry via a tooltip, by hovering over the entry with the mouse.

Figure 3: Tooltip showing the logical processor ID

A major benefit of a heat map is that it scales really well to large data sets. The new Task Manager will show as many logical processors as the OS supports (up to 640!). To make sure you always see the information at a meaningful size, when the data set gets too big for the window, the heat map scales to best fit, and a scroll bar appears as needed.

Figure 4: With 160 logical processors, the CPU graph scales using a scrollbar

For those of you who really like to (micro-) manage every last detail of your system, you can even set which logical processor(s) each of your processes can use. To do this, you first find the ID of the logical core by hovering over one of the cells in the heat map, then go to the Details tab, right-click the process you want, and click “Set affinity.”

Figure 5: Set process affinity from the Details tab

Figure 6: Select the logical processors for the process

Of course, setting processor affinity is only for the super-technical user who has a need or desire for that level of control—you can severely affect your system’s power management and performance if you don’t know what you are doing—so we made sure the OS would do a great job taking care of this for you. It is hard to do better than the sophisticated algorithms that Windows uses to automatically manage which processes are allocated to each logical processor based on hardware capability and topology.

--Ryan Haveson

Optimizing for both landscape and portrait

Thu, 20 Oct 2011 19:00:00 GMT -

As we have demonstrated Windows 8 in many forums, we've tended to use landscape orientation (widescreen) quite a bit. Primarily that's because often we're projecting, and it makes for a better experience that way. Another reason is that many of the early devices (such as the Samsung tablet issued at //build/ with Windows Developer Preview) are widescreen, which is ideal for showing side-by-side applications using the new Snap feature, and that tends to work well in landscape. We have done a ton of work to enable a fast and fluid experience in rotating the screen, and a great experience for people who prefer the portrait orientation, and as you will read, this is heavily influenced by our experience studying what factors contribute to a preference in either orientation. We even did work on our Visual Studio and Expressions tools to make sure developers have great tools support for building applications that work well in both orientations. David Washington on our user experience team authored this post on landscape and portrait screens in Windows 8. He also offers session APP-207T from //build/. --Steven

A Windows 8 PC is really a new kind of device, one that scales from small, touch-only tablets to laptops and desktops. While reimagining Windows 8, we designed it to deliver a great experience regardless of the form factor or screen orientation. Tablet devices allow for ergonomic flexibility, allowing you to hold the device in whichever orientation is most comfortable to you and best suits your content.

One of the best things about a tablet is that you can hold it in your hands. It’s personal. Whether you’re reading the Sunday newspaper or browsing through a stack of wedding photos, being able to hold and touch what you interact with ties you emotionally to it. In the digital age, a lot of what is most important is on devices, so when planning Windows 8 we wanted to make sure that the experience could support any orientation that the device could be held in.

As we designed the end-to-end experience on different form factors in Windows 8, we used the following principles:

• The experience tailors itself for all form factors: small screens, wide screens, laptops and desktops.
• The experience takes advantage of widescreen formats for multi-tasking and for full-screen video.
• The device can be held and interacted with in the way that is most comfortable.
• Developers have the opportunity to create one app that runs on all views and orientations across form factors with minimal effort.

We spent a considerable number of hours studying people as they used tablet devices in our usability labs as well as in their own homes. We’ve watched people who were already familiar with tablets, and we’ve watched people new to these form factors get started, and we stayed in touch with them for months afterwards. We noted grip styles, body postures, hand movements, and interactions with a variety of apps, device placements, and orientations. We saw a wide spectrum of variability, and we listened to users identify the factors influencing their choices about body and device orientation.  Influences on these choices included anthropometric factors like hand size and thumb reach distances, ergonomic factors like repetition and fatigue, hardware factors like accessibility to hardware buttons, environmental factors like where they were using the tablet (e.g. in the kitchen, bedroom, or living room), and physical factors like if they were using it standing, sitting on a couch, or sitting at a desk.  The number of combinations was staggering, contributing to the basic conclusion that postures, grips, and orientations change fairly frequently.  Simply put, there’s no one way to hold a device and people naturally seek to find a comfortable position and orientation that feels right for what they are doing with the device at the time.

We initially thought that landscape or portrait orientation was mostly influenced by personal preference.  Each person that we watched rotated the device and each expected the device and UI to work with them at that moment.  What was surprising was that as people become more familiar with the device and the apps they cared about, the most unique influencer on whether they rotated the device was the type of content on screen. If the content and experience felt better in landscape, people naturally used the app in landscape-mode.  If the content and experience felt better in portrait, the app was used in portrait. As an example, most people prefer to watch a movie full-screen in landscape without black bars, while they prefer to read a long article or webpage in portrait, as it requires less scrolling.  The preferences we heard people express were heavily influenced by their own sense of whether the app offered a great experience in that orientation.  We’ve received questions and feedback about whether Windows 8 is “landscape first” or “portrait first.” Our point of view is that both portrait and landscape orientations are important, and experiences can be great in either orientation. Rather than picking a posture and orientation for optimization, we designed an experience that makes sense regardless of how the device is held, one that feels tailored for the app and its content.

Our goals when looking at landscape and portrait were as follows:

• You can easily rotate your tablet to best suit your task or ergonomic posture.
• Rotation in Windows is fast and fluid.
• Windows rotates predictably across the system and apps – keeping the user in control.
• Developers can easily build high quality and intentional landscape and portrait layouts, depending on the experiences they want to enable.

Windows in landscape mode

Some people have asked why we showed so much of the Windows 8 user interface in landscape at the //build/ conference. Windows 8 is a reimagining of all PCs, and it’s not just for tablets. It will run on hundreds of millions of laptops and desktops (designed for Windows 7 and new for Windows 8), many of which are and will be landscape-only. Also, in landscape and widescreen, we can offer a multitasking experience (snapping two apps side by side) and full-screen video playback without letterboxing. (In addition, for many of our larger demonstrations we are projecting to huge screens, which look better in landscape).

We’ve designed Windows 8 to be ergonomically comfortable in all orientations. We found that a comfortable posture for using a tablet in landscape is to hold in both hands and touch the screen with your thumbs. For this reason, we’ve designed the majority of the experience to be easily accessible under your thumbs. We also optimized the system to scroll horizontally, which feels fast and fluid in landscape as well as in portrait mode.

Windows in portrait mode

Windows is also designed to work great in portrait mode. We studied extensively scenarios like reading news in the web browser, looking at portrait photos, and scrolling through long lists of email messages, and we folded what we learned into designing a system that works seamlessly across orientations. We tuned the system experiences like the keyboard, file picker, and charms to work great in portrait as well as landscape. We wanted to make it so you don’t need to relearn the system when you switch to portrait mode; it just works.

Rotation

Because one of our goals was to make the rotation transition between landscape and portrait feel fast and fluid, teams across Windows put significant work into streamlining the path of this transition, from the accelerometer hardware up through the graphics stack to the app.

An important part of the transition between landscape and portrait is the animation. The animation choreographs the appearance of a smooth transition between the two layout states. The timing of the animation is important as it had to be tuned to feel fast and responsive, while remaining smooth enough to make the transition not feel jarring. The Desktop Window Manager (DWM), which is foundational to the smooth animations in Windows 7 and Windows 8, orchestrates this animation.

We are continually working to make the rotation as stable and predictable as possible, as we know how annoying an over-eager rotation can be. Before the rotation starts, the system waits for the accelerometer to stabilize to prevent accidental rotations. We also wanted to keep you in control of the rotation experience, so that it wouldn’t be triggered accidentally. We introduced a hardware orientation lock to “override gravity” and keep the orientation the way you want it.

Different screens

In a future post, we’ll cover in depth how Windows 8 scales to different screens from a developer perspective, but it’s worth talking a little bit about screen size in the context of landscape and portrait orientations. Windows 8 will run on PCs with different sized screens and with different resolutions and different aspect ratios; from 4:3 screens that are closer to square, to widescreen 16:9 screens and everything in between. Our scaling platform enables Windows and apps to seamlessly adapt and reflow content to of these different screens and make use of the space. You can use many of these devices in either portrait or landscape. This diversity of choice is unique to Windows. You can choose the device and the orientation that best suits your needs and usage.

The minimum resolution that Windows 8 Metro style apps can run is 1024x768. We chose this size as it is a common size designed for use on the web, and a strong majority (i.e. 98.8%) of Windows users can run at this resolution or higher (see chart below).

We think it is important to have a minimum resolution for apps, as it allows all developers to design the smallest main view of their app without fragmentation across devices. This minimum also ensures that users do not see broken app layouts due to small screen sizes.

The resolution that supports all the features of Windows 8, including multitasking with snap is 1366x768. We chose this resolution as it can fit the width of a snapped app, which is 320px (also the width designed for many phone layouts), next to a main app at 1024x768 app (a common size designed for use on the web).

These limits will be enforced at runtime. We are breaking from past practices in not providing a workaround for this, given that the main motivation is to make sure Metro style apps are designed to function fully at a specific published resolution. In the Developer Preview we did not have any useful runtime warning beyond the communication on the download site, which we will of course address for the beta. We did notice that a set of folks running on 600x800 virtual machines or 1024x600 Netbooks were inconvenienced and we're sorry about that. It is worth noting that the runrate of 1024x600 machines is very low as screen resolutions on the low end have moved to 1280x800, which supports Metro style apps without snap. This resolution is still a distant second to fully capable resolutions and we expect to see the shift to higher resolution screens continue as new PCs are brought to market.

Rotation and developers

In Windows 8, apps power the experience, so we worked to make it as easy as possible for developers to build both landscape and portrait views. As on any other platform, developers can choose which orientations they support for their apps, and how their experience is tailored. We expect most developers will provide a landscape view, as existing laptops and desktops make up the highest number of PCs in the marketplace today. However, if an app’s experience could support both landscape and portrait, supporting the portrait view is just an incremental amount of layout work.

Using the same techniques used to build for the Snap feature or for different sized screens, developers can easily build portrait experiences. HTML5 developers will use CSS media queries to bind their layout style to the orientation of the system, whereas XAML developers will change their layout in response to view state events. In both HTML and XAML, all of the adaptive controls and templates that the platform provides will support both portrait and landscape. Additionally, the system automatically handles the transition animation without any additional effort from developers. If there is content in an app that is best suited for one orientation, developers can prefer one orientation or the other, and the system will keep the app in that orientation (if the device supports it).

For testing apps, Visual Studio 11 and Expression Blend allow developers to test their apps in portrait and landscape mode and on different screen sizes and aspect ratios, even if they don’t have access to a tablet device.

You can choose the device that best suits your preferences, pick it up in the way that is most comfortable, and the experience accommodates that posture. Apps can take advantage of widescreen with multitasking, and still look great in portrait orientation with minimal additional work.

Here’s a short video to show you the landscape-to-portrait transitions in action.

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

We look forward to you trying it out!

Thanks,
David

Designing search for the Start screen

Tue, 18 Oct 2011 19:00:00 GMT -

Given the ton of interest in the design of the new Start screen we wanted to dive deeper into the topic of search. There's a clear focus on efficiency and overall professional productivity in the comments. For professional scenarios, every keystroke matters. One new aspect of the Windows 8 platform is the ability for Metro style apps to deliver a customized search "contract." For this post we'll focus on the built-in search capabilities for files, settings, and apps, which update the Windows 7 search features. You can learn more from our //build/ session on search, which provides a detailed look at the topic of this post. With that lens, Brian Uphoff, a program manager on our Search, View, and Command user experience team, authored this post.
--Steven

In our previous related posts (Evolving the Start menu, Designing the Start screen, and Reflecting on your comments on the Start screen) we discussed the evolution of the Start menu and the reasoning behind the design. We also discussed how organizational mechanisms and search are powerful tools that make it easier to find and launch apps. As you install more and more apps, these tools become increasingly important. For the past several releases, searching from the Start menu has been established as the quickest way to find and launch apps, particularly for keyboard users.

When planning Windows 8, we wanted to make sure the efficiency and dexterity of the Windows 7 Start menu search was carried forward into the new Start screen. Before we dive into the details of the new experience, let’s take a quick look at the evolution of search from the Start menu, and how people are using it today.

Evolution of searching from Start

The search box in the Start menu as we know it today first made its appearance in Windows Vista. It became easy for users to search for programs or apps, settings, and files on the desktop and in personal folders like Documents, Pictures, Music, and Videos. The search experience aggregated different types of results in one view with programs and settings combined in a single group. The results of a query displayed a small set of items in heuristically sized groups. You needed to click “See all results” to see the rest in Windows Explorer, which aggregated everything into one ungrouped and unsorted view.

Figure 1: Start menu search in Windows Vista

In Windows 7, we expanded results to include detailed Control Panel tasks in addition to the main Control Panel pages. We also separated out Control Panel items from programs into a unique group that allowed you to more easily focus on the type of result you were looking for.

The overall experience aggregated different types of items and had a fixed limit on the number of results that could appear. This was because the result set was limited to the size of the Start menu. Clicking a group header took you to Windows Explorer for programs and files or to Control Panel for settings. Each experience had a type-specific view, though the search results order diverged from what was shown in the Start menu. Showing an aggregated view in the Start menu required compromising on performance in addition to space because we would search all programs, Control Panel items, and files, even if you were looking for only one of these data types.

Figure 2: Start menu search in Windows 7

When we look at the usage data of how people are using the Start menu to search in Windows 7, it’s clear that searching to launch programs is the most frequent and important activity users engage in with Start search.

Our telemetry data shows that 67% of all searches in Windows 7 are used to find and launch programs. Searching for files accounts for 22% of all Windows 7 Start menu searches, and searching for Control Panel items about 9%. Searching for email messages via Start Menu is very rare (less than 0.05%). The remaining 2% are searches executing the “Run” functionality.

Figure 3: Windows 7 Start menu search usage data

Searching from Start in Windows 8

Searching via the Start menu has continued to evolve with each release. The Windows 8 Start search experience builds on top of search features available in Windows 7 and provides a unique view for each of the three system groups - Apps, Settings and Files. These search result views are a natural progression from the Windows 7 groups and are easily accessible from anywhere in the operating system via the Search charm or keyboard shortcuts. Separating the search results into views means we can tailor the experience for each data type. For example, the File search view provides you with filters and search suggestions while typing to quickly complete your query.

In Windows 8, we expect people will be acquiring and installing more apps than ever before. Had we continued using the Windows 7 Start menu search interface to search for a Control Panel item, you would always see app or program results before Control Panel results, displacing many Control Panel items from being the first match. This and other constraints on the existing design required us to develop a new approach—this is especially true as we consider the increasing use of larger monitors or higher DPI screens where longer menus become even more difficult to use and navigate. In Windows 7, the total number of results that could be shown in the Start menu was limited. Depending on the number of groups with matching results, an average of 3-4 results were shown per group. Very rarely did all results for a group show up, and the organization of the results was pretty unpredictable.

With Windows 8, on the other hand, we’re following an app-first model, where each app developer understands their data and users best, and knows the best way to present the information to them. Using the same model for search, we believe that always having a quick and consistent way to get directly to settings or file search results gives you precision and control over the type of results you’re looking for. In Windows 8, each view is tailored for the type of content you’re searching for, and shows all the results, instead of limiting them due to screen real-estate.

One change a few of you will notice is that file search results no longer include email messages and contacts. The inclusion of email search never got the generalized support from mail clients that we had hoped for, though at least one mail client did support it (one reason why email searches are rare in the Start menu <0.05% of total searches). With the app-first approach in Windows 8, Metro style email apps will use the search contract to provide a rich set of filtered search results in a view customized for email. In comparison, email clients and other apps in Windows 7 have no control over how their search results are presented.

We paid special attention to ensuring the number of keystrokes required to find and launch apps, settings, or files is at parity with or better than in Windows 7. We’ve introduce a set of keyboard shortcuts to help users quickly and efficiently get to settings search results (WIN key + W) or file search results (WIN key + F), thus reducing the total number of keystrokes needed to find and launch settings or files. We’ll cover how we maintained and increased keyboard efficiency across these views in more detail later in the post.

Searching apps

App search results show the full set of apps (both their “friendly” names and executable names) for which the search term matches the name. As the number of installed apps increases, it becomes difficult to browse through a large list to find an infrequently used app. Search helps quickly filter and reduce a large list of apps down in just a few key strokes. We wanted to make sure we preserved the same keyboard usage patterns as Windows 7. You don’t have to first click on the Search charm to begin searching – simply start typing in the Start screen and you’ll see your list of apps filter down to the one you are looking for.

Figure 4: Full-screen app search results

Also note that the Most Frequently Used (MFU)-based ranking of app search results from Windows 7 is preserved in Windows 8. For example, if you type “paint” in the developer preview you get 2 apps back as search results – PaintPlay and Paint. If you predominantly just use Paint, it will be ranked higher than PaintPlay as you use it more often. So, launching Paint (or other apps you frequently use) becomes more efficient the more you use app search.

Some of you have pointed out that many users won’t discover that they can simply type to start searching in the Start screen. Search is closely associated with typing— the most common pattern to search in the Start menu is to bring up the Start menu by using the Windows key or by clicking the Start button and typing. That exact and efficient behavior is preserved in Windows 8 as we have observed and found that pattern is what users care about most. Our experience in user tests, and even when people at //build/ tried the Develop Preview for the first time, shows that people tend to serendipitously discover this feature early in using Windows 8, and so we're confident it will not be a hindrance to usability. Nevertheless the Search charm is highly visible, and selecting it shows the Search box.

The Windows 7 Start menu also included “Run” functionality for commanding and navigating Windows. This has been carried over to Windows 8 as well—tasks like running scripts and .exes in the user’s PATH are still possible and supported in App search. Search continues to support launching folders in Windows Explorer by typing in full paths. For example, typing “C:\” in Start search results in the set of folders in the C: drive appearing below the search box. Pressing the Down Arrow key moves selection through the list and autocompletes the folder name in the search box, allowing users to continue typing to further refine the path. You can do the same with UNC (\\foo\example) paths as well. And of course WIN key + R will switch to the desktop and bring up the classic Run dialog, just as you would expect.

Figure 5: Typing a path into Start search

Searching settings

The settings search experience brings together all settings and Control Panel items across the system in one view. Settings search results are matched not only to the name of the Control Panel applet or task, but also to the various keywords that may describe it. We have also heard your frustration that shutdown is not available as a search result, and we will address this along with improvements to the Start user interface for shutdown (as a reminder, you can also just use the power button or close the lid).

Figure 6: Full-screen settings search results

Searching files

The number of files on PCs keeps increasing over time as users continue to acquire and create more documents, music, photos, and videos. Our goal, while redesigning the file search experience, was to make it seamless and complete so you can achieve your task of quickly finding a file without having to transition to Windows Explorer.

In File search, you’ll also see search suggestions as you type to help you quickly and efficiently complete the search. The indexer provides these search suggestions based on the content and properties of files it knows about. Search suggestions are a very powerful concept made popular and used extensively on the web—they help you to pinpoint relevant search terms with just a few keystrokes. In Windows 8 we built search suggestions into the file search experience and also made this feature available in the platform for all Metro style apps to use. Note, this feature also accounts for typos or spelling errors, and suggests the auto-corrected search term as you type. Using the arrow keys to choose suggestions autocompletes the term in the box. This makes it easy to add more terms to the query and quickly narrow down the set of results to find the one you want.

You can also still search using AQS (Advanced Query Syntax) from Windows 7. AQS allows for greater precision and control when constructing the query to get targeted results. Here are some sample searches and their advanced query syntax:

Query	AQS Syntax
Find all files authored by Brian or David	author: (Brian OR David)
Find all photos with an F-stop of 2.8 where no flash was fired	f-stop:2.8 flashmode:no flash
Find all files where the file name contains a word starting with Metro and the file size is greater than 1MB	filename:$<Metro* size:>1mb

Figure 7: Search suggestions based on the content and properties of files

Figure 8: Full-screen file search showing results

Separating searches for apps, settings, and files into their own views allows room for each of them to evolve and breathe— this way they can each provide their own ideal display format—unlike the single list of results in previous versions, which required conformity to achieve aggregation in the limited space. For example, the file search view also provides filters to easily refine the results based on the type of file you’re searching for. Filtering by type is a powerful way to efficiently reduce the results set, irrespective of where the file is saved.

More relevant and contextual information for each file is also now displayed to make the search experience complete. This helps differentiate between similar results and also makes it clear to you why a given result was returned, by highlighting the property that matched the search term—something not possible in the Start menu before. For example, when searching for the term “performance,” the results now highlight where “performance” matched. In one result, it matched on the Title property, and is clearly indicated that way in the result. The results also show file type and size to help further disambiguate between results.

Figure 9 File search results highlight the property that matches the search term

Using a mouse, hovering over a result reveals a rich tooltip with some additional details. For example, for the video result shown below, the rich tooltip shows the duration of the video, frame height, frame width, date modified, and the full path to the file. Using touch, pressing and holding on an item reveals the tooltip.

Figure 10: Rich tooltip reveals additional details in file search

Designing Start search for dexterity

Designing for efficiency and dexterity is a core goal of the Start search feature team. As such, using the keyboard to launch apps, settings, and files from search is a very important part of the Start search experience. We also put a lot of thought into preserving existing keyboard patterns, which both average and advanced users have come to rely on, and have built muscle memory around.

Our telemetry data shows that many users leverage the Start menu as a means of commanding Windows. They use specific key-combinations to efficiently launch apps. For example, pressing the WIN key, typing “calc”, and pressing ENTER launches Calculator. Many advanced users know that typing “cmd” and then CTRL+SHIFT+ENTER opens an elevated command, and that typing “notepad c:\mynotes” creates or opens a .txt file. If you watch the keyboard demonstration from the //build/ keynote, you will see many of these used.

These keyboard patterns continue to work in Windows 8 just as well as in previous versions. Pressing the WIN key takes you to the Start screen. Simply start typing in the Start screen and the Search pane automatically opens with the search term in the search box, and the view filtered to show apps that match the term.

The fastest way to search settings and files from anywhere in the system is to use a set of keyboard shortcuts introduced to increase efficiency. These Windows 8 shortcuts reduce the number of keystrokes needed to launch a setting or file to a number equal to (or less than in many cases) what was required in Windows 7. Alternatively, you can also use the Search pane, which indicates the number of results matching the search in each view, to switch between apps, settings, and file searches.

and type	Apps search
+W	Settings search
+F	Files search

Figure 11: Windows 8 Start search keyboard shortcuts

Based in part on the feedback here, we are working on a change that we hope to have available in our beta release, which will take you directly to app search results when you select the Search charm in the desktop.

The efficiency of using the keyboard doesn’t stop at just typing to start a search. Sometimes the app, setting, or file that you want to launch is not the first result shown. You can use the arrow keys to quickly move down to the desired app in the results list, and then press ENTER to launch it. The white box that shows keyboard focus tells you which app will be launched on ENTER. This enables you to efficiently launch any app, setting, or file matching a search. In Windows 7, you could only launch one of the top 3-4 results with this kind of efficiency.

When we looked at some of the common Control Panel items people were searching for (for example, searching for power options using the term “power”), it quickly became clear that because we favored app results first in our Windows 7 design, “Power options” was the fourth result, below all the power shell app results. If you installed Office and frequently used PowerPoint, you saw PowerPoint along with Power shell (32 bit, 64 bit, and the Help file) ahead of ”Power options.” Extending a similar design in Windows 8 would have meant that the position of the “Power options” result would continue to fluctuate as you installed more apps on your system. This forces you to scan through increasingly large result-sets every time you search for a particular app or setting or file.

In Windows 8, we also provide the count of results per system view, so you immediately know how many apps or settings or files match the search term. Switching search views is also designed so you can easily switch views without taking your hands off of the keyboard. In the example shown below, to switch to settings search, press the Down Arrow key and the focus shifts to settings in the search list. Press ENTER and you will see settings search results. As mentioned before, you can continue to use the arrow keys to choose the desired item and press ENTER to launch it. Pressing TAB allows you to quickly switch between the search results list and the Search pane.

Figure 12: No results for apps – but use the arrow keys to switch to settings view, which shows 17 results

Figure 13: Use the arrow keys to choose a settings search result

To add to our earlier point on preserving search efficiency, here are some comparisons of the number of keystrokes for launching frequently used apps via search. In Windows 7, you would press the WIN key, start typing in your search term, and then press ENTER to launch the program. We count all the keystrokes end to end. In Windows 8, you can apply the same pattern for searching for apps (WIN key, type in the search term, press ENTER to launch). Launching Word, Calculator, Paint, or Media Player by pressing the WIN key and typing "word", "calc", "calculator", "paint", "player", or "media" in the search box takes precisely the same number of keystrokes in Windows 8 as it does in Windows 7.

To launch settings in Windows 7, you would press the WIN key, type in the query, arrow down to the result you wanted, and press ENTER to launch it. In Windows 8, you can use WIN key + W to launch settings search, type in a query, and press ENTER to launch. Typing WIN key + W and typing "uninstall", "device manager", or "defender" gives you the same results with precisely the same number of keystrokes in Windows 8 as in Windows 7. In some cases, it takes even fewer keystrokes than in Windows 7 (for example, pressing WIN key + W, typing “power” and then pressing ENTER to launch power options).

This dexterity-focused design isn’t all we have done to make search more efficient. We have also made key performance investments across the system. In current testing of Windows 8, our search performance improvements have cut app search time in half for desktops and laptops. The improvements are even larger on netbooks.

Figure 14: Performance comparison showing % decrease in execution time of app search

Designed for touch, too

We discussed the details of designing search for dexterity while using the keyboard, but this design works equally well for touch. To begin searching in Start, simply swipe the edge and tap on the Search charm. This opens the full list of installed apps. You can use the touch keyboard to search for a program to launch, but you can also use semantic zoom to zoom out, and then tap on the section that contains your app. Start search is lightweight, fast, fluid, and quickly gets out of the way as you pan through your list of apps, settings, or files.

The Search pane makes it easy to continue searching for the same term in other system views or Metro style apps with just one tap. Touch-friendly search suggestions minimize typing on a touch screen, and the search contract provides a framework for search suggestions that developers can use for their own Metro style apps as well. In addition, we designed touch-friendly filters with result counts in the file search view to help users quickly refine the search results set.

The new Start search experience makes it easier than ever to search for content in your PC or in apps from anywhere in the system. It’s been designed to work seamlessly and efficiently across the range of devices that Windows will run on, and across different input mechanisms such as the mouse, keyboard, and touch. Start search brings apps, settings, and files together with other Metro style apps that implement the Search contract, creating a unified and consistent search experience. We will talk more about the search experience and the search contract in a future post. In the meantime, you can get more information from our talk at //build/ on the Search contract.

Here’s a video showing how easy and efficient it is to quickly launch apps, settings, and files from anywhere in the system using the keyboard:

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

We look forward to your continued feedback as you try out the Windows 8 Start search experience!

Thanks,
Brian

The Windows 8 Task Manager

Thu, 13 Oct 2011 19:00:00 GMT -

As we mentioned during the Windows 8 keynote at //build/, every 15 years or so we choose to update Task Manager. Of course that was said in jest as we have incrementally improved the utility in just about every release of Windows. For Windows 8, we took a new look at the tool and thought through some new scenarios and a new way of tuning the tool for "both ends of the spectrum" in terms of end-users and those that need very fine-grained control over what is going on with their PC. Ryan Haveson, the group program manager of our In Control of Your PC team, authored this post. Note: This post is about Task Manager, not about closing Metro style applications :-)
--Steven

We are really excited to share some of the improvements we are making to the Task Manager in Windows 8. Task Manager is one of the most widely used apps, and it has a long history. It showed up in early versions of Windows as a simple utility to close and switch between programs, and has had functionality added to it through several releases to make it what it is today.

Figure 1: Windows 3.0 Task List

Figure 2: Windows NT 4.0 Task Manager (now with “new task”)

Figure 3: Windows XP Task Manager (with new Networking and Users tabs)

Figure 4: Windows 7 Task Manager

Because Task Manager is so widely used, we knew that any changes we made would be noticed, so of course we were both excited and cautious about the effort. At the beginning, there were a few key problems that we knew we wanted to address:

• Build a tool that was well designed, thoughtful, and modern. After all, even a technical tool can benefit from a focus on design.
• Fill some of the functionality gaps that drove some of our most technical customers to use other tools such as Resource Monitor and Process Explorer.
• Organize and highlight the richness of data available to make it more elegant and clear for those who want access to a new level of data.

How do people use Task Manager?

To really make Task Manager great at what it currently does, we wanted to first understand how people were using it. Over the years, it had grown to support many different scenarios. As of Windows 7, you could use Task Manager to close applications, to find out detailed data about your processes, to start or stop services, to monitor your network adaptor, or even to perform basic system administrator tasks for currently logged on users. That is a lot of functionality.

Because of the investments we made in telemetry, we had some pretty good data to start with. We combined this with individual customer interviews and observation in the research lab to understand what people were doing with Task Manager and why they were doing it.

Figure 5: Which tabs are people using?

This data is pretty interesting. What it shows is that people are spending most of their time using the first two tabs, which are pivoted around views of applications and processes. Although it is not surprising, it was interesting to see that the usage was roughly evenly split between the Applications tab and the Process tab. This indicates that there must be some significant detail lacking in the Applications tab, which is causing people to go to the Process tab. So, next we looked at how people were using the Process tab to understand what they were doing there.

Figure 6: Many users sort the process view on resource usage

When we looked at this data, and then correlated it with interviews and observations of users in our research labs, we found that people were using the process tab either to look for something that was not on the applications list (e.g. a background or system process), or to see which processes were using the most resources.

So next we looked at what actions people take in Task Manager.

Figure 7: The goal is often to close or “kill” an app or process
Click to view a larger version of this chart

Looking at the data and talking with customers, we determined that the most common usage of the tool was to simply end or “kill” an application or a process.

Goals of the new Task Manager

Based on all of the data and our background research, we decided to focus energy on three key goals:

• Optimize Task Manager for the most common scenarios. Focus on the scenarios that the data points to: (1) use the applications tab to find and close a specific application, or (2) go to the processes tab, sort on resource usage, and kill some processes to reclaim resources.
• Use modern information design to achieve functional goals. Build a tool that is thoughtful and modern by focusing on information design and data visualization to help achieve the functional scenario goals.
• Don’t remove functionality. While there are some notable core scenarios, there is a really long list of other, less frequent usage scenarios for Task Manager. We explicitly set a goal to not remove functionality, but rather to augment, enhance, and improve.

A key issue we intended to address was how we could add all of the interesting new functionality without overwhelming users. To solve this, we pivoted around a "More/Fewer details" button similar to the new copy file dialog model.

Figure 8: Fewer details view

Figure 9: More details view

This model allowed us to optimize the default view (“Fewer details”) around the core scenario of finding an application and closing it. It also allowed us to add much more detail in the other view because it would only show up when someone asked for it. In the “More details” view we decided to stay with the existing tabbing model of Task Manager and focus on improving the content of each of the tabs. This would help us to augment, enhance, and improve what we already had, without removing functionality.

Scenario #1: Ending processes quickly and efficiently

We know from many third-party tools (or tools like sysinternals Process Explorer) there are many things we could add to Task Manager for power users, but we knew we had to first address the mainstream users because we didn’t want to create something that would overwhelm the majority of our customers. We will of course continue to value third-party tools as they allow for specialization and unique innovation around this and many tasks. For the default view, we designed a minimalist experience that appeals to the needs of the broadest customer base and most common scenario. When you launch Task Manager for the first time in Windows 8, you see a very clean view of your running apps. We made the default view great at one thing: killing misbehaving apps. And we removed everything that did not directly support that core scenario.

 

 Figure 10: Default view of Task Manager in “Windows 8”

The value of the default view is all about what we took out. We removed everything not focused on the core task of killing apps, which makes the design focused and efficient. Specifically:

• We took out the tabs from this view, since they distract from the core scenario.
• We removed the menu bar from the default view.
• This view shows just the apps, and removes individual windows that can’t be killed anyway.
• We took out things that clutter the experience, such as resource usage stats and technical concepts that most users don’t understand.
• No double prompts. If you click ”End task” we don’t ask you, “Are you sure?”, we just kill the app, and quickly! (But be careful, because we also won’t prompt you to save!)

Check out how much cleaner and more focused the new Task Manager is compared to the Windows 7 Task Manager with the same applications and windows opened:

 
Figure 11: Windows 7 and Windows 8 Task Managers, compared

After taking out all of the extras, you are left with a tool that is great at one thing: killing a misbehaving app. And this is perfect for many users who are experiencing the pain of a “not responding” app that won’t go away using the app’s Close button.

Scenario #2: Diagnosing performance issues

A lot of what is new with Task Manager is shown only when you go to the “More details” view. This is the realm of the power user, so keep in mind that mainstream users may never want to get into this level of detail, and all of their needs should be met by the ”Fewer details” view above.

Here is what you will see in this new view:

 

Figure 12: The new processes tab and the heat map

The heat map

The most noticeable difference in the new processes tab is the new heat map, which represents different values with color. Our telemetry data told us that it was very common for users to go to the process tab, sort by CPU or memory utilization, and then look for applications consuming more resources than expected. The nice thing about a heat map is that it allows you to monitor anomalies across multiple resources (network, disk, memory, and CPU utilization) all at the same time, without having to sort the data. It also allows you to find the hot spot instantly without needing to read numbers or understand concepts or specific units. In usability studies we used an eye-tracking system to test what users looked at when presented with various ways of visualizing this information. This helped us narrow our choices to a design that efficiently draws user’s eyes to the most significant resource problems. Below you can see the eye movement of a participant in one of our eye-tracking studies overlaid on top of a screen shot of what he was looking at. The red dot indicates a place where his eye paused, and the lines show where his eye had quickly moved from previously.

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

 

Network and disk counters

Many power users supplement their usage of Task Manager with other tools such as Resource Monitor simply because in the past Task Manager did not show per-process network and disk attribution. This was a gap, when you consider that a spinning disk or multiple applications competing for network bandwidth are the root cause of many perceptible PC performance issues. The new Task Manager now shows these resources at the same level of detail as memory and CPU.

Lighting up the resource usage

One of the biggest causes of PC performance issues is resource contention. When a particular resource is being used at a rate above a threshold number, the column header will light up to draw your attention to it. Think of this as a warning indicator, letting you know a good place to start looking if you are experiencing performance issues. Below, you can see that the CPU column header is highlighted to draw your attention to the fact that you may have multiple applications competing for CPU time.

Figure 13: Resource usage indicators

 

Grouping by applications, background processes, and Windows processes

A big challenge with today’s Task Manager is that it is hard to know which processes correspond to an application (apps are generally safe to kill), which are Windows OS processes (killing some of these can cause a blue screen), and which are miscellaneous background processes that may need to be explored more deeply. The new Task Manager shows processes grouped by type, so it is easy to keep these separated while still providing an ungrouped view for situations where you need it.

Figure 14: Grouping by process type

Friendly names for background processes (and services, and everything else)

Looking at the screen shot above, do you see the line item for "Print driver host for applications"? In the old Task Manager, this showed up as “splwow64.exe”.

But if you still want to see the executable name, of course you can add it back as an optional column.

Grouping top-level windows by app

One of the most distracting parts of the old Task Manager is that the Applications tab was a flat list that included all of the top-level windows from all processes in the system. While the list of top-level windows is interesting information to have, it is often overwhelming to look at and sometimes a single window cannot be killed without closing all the other windows for that process. To address this, the new Task Manager now groups top-level windows under their parent process. It allows for a much cleaner view for typical usage, helps you focus on killable processes, process resource usage, and allows you to see which windows are owned by each process so you know what will be closed if you kill it.

Figure 15: Grouping top-level windows by process

What’s a fussvc.exe?

Have you ever looked through the process list, seen something like “fussvc.exe” and wondered what it was? Adding friendly names was a good first step to resolving this problem (fusssvc.exe is actually the Fast User Switching Utility Service), but of course, to really find out what this process is, you need to search the web. The new Task Manager integrates a search context menu on right-click, so you can go directly to your default search engine (which you can customize) to see more details and relevant information. This can make a huge difference when deciding whether a background process is doing something useful or just wasting cycles.

Figure 16: Search the web for details on obscure processes

 

Figure 17: Search results for “fussvc.exe Fast User Switching Utility Service”

Service host details and friendly names

If you open up Windows 7 Task Manager to the Processes tab and select “Show process from all users,” you will probably see eight seemingly identical instances of “svchost.exe”. This is one of the most commonly noted "not very informative" sources of information we provided. Of course, some of you know that this is really just a service host process and you can add the PID column, go to the services tab, sort by PID, see which services correlate to that PID, and then reverse-look-up friendly names for each of the services… but that is a lot of work (and not everybody knows this)! With the new Task Manager, we show all of the services grouped by process with friendly names for each of them, so you instantly can see what is going on when an instance of svchost is consuming a lot of resources:

Figure 18: Service host grouping and details

As you can see, we added quite a lot to the new Task Manager (and we only showed you the first tab!). Task Manager was a unique opportunity for user experience designers and researchers working together with technical program managers and engineers to create a clean, organized, and efficient design. We made it more streamlined for mainstream users, and more detailed for power users.

I will leave you with a quick demo where you can see what it looks like in action.

Download this video to view it in your favorite media player:
High quality MP4 | Lower quality MP4

-- Ryan

Reflecting on your comments on the Start screen

Tue, 11 Oct 2011 21:00:00 GMT -

We've been having a lot of discussion regarding the two recent posts on the Windows 8 Start experience. Those of you who have used the Developer Preview are contributing to our understanding of your individual usage patterns and what is easier or more difficult than in Windows 7. As a reminder, we released Windows Developer Preview build with the full product "enabled" even though we still had much feature work to do in the user interface. We did this in order to foster the dialog and we want folks to understand that the product is not done. We've seen some small amount of visceral feedback focused on "choice" or "disable"—a natural reaction to change, but perhaps not the best way to have a dialog leading to a new product. We’re going to focus this post on making sure we heard your constructive feedback around the design as we continue to evolve it. Marina Dukhon, a senior program manager lead on the Core Experience team, authored this post focused on specific comments and the actions we are taking based on what you have said. --Steven

On behalf of the team, I want to thank everyone for their active engagement on the Start screen blogs over this past week. We have been following all of the comments and responding as much as we can. We know major changes like this can be controversial and we are looking forward to continuing this dialog with you. I wanted to address some of the specific topics that have been brought up so far as they pertain to the design. I know this doesn’t address all of your questions, but rest assured that we are listening and will be continuing this ongoing conversation.

Does the data support all customers?

@Andrew wrote:

"I'd like to point out that this data you collect is most likely from non-corporate users, you're basing all your statistics around home users and not business users. Most enterprises will turn off the CEIP by default in Group Policy as a security precaution and to prevent chatter from the network."

Andrew, while it’s true that some enterprises choose not to enable the CEIP (Customer Experience Improvement Program, which gives us anonymous, opt-in feedback about how people are using Windows,) we still receive a huge amount of data from this program, including from enterprise customers. In addition, knowing the region, language, edition, and deployment attributes of the product allows us to further refine the data as needed. We often refer to this data as a full "census" (again noting that the data is opt-in and anonymous) as the number of unique data points is magnitudes beyond a "sampling."

In addition to the CEIP program, we have a wide variety of channels to our corporate customers to understand their needs. For example, we collect feedback continuously during direct engagement with customers (such as during on-site visits and in our briefing centers around the world), from advisory council and early-adopter program members, and at public events such as TechEd and //build/. We also work closely with industry analysts (via consultations and their research) and execute a wide range of our own research studies directly. From these interactions, we know the kind of functionality and control that enterprises want over the Start menu and we are definitely taking these into account as we are designing and developing the changes for Windows 8.

When you look at the data, we can see that enterprise customers do, in fact, have some different experiences with their Start menus:

• While 81% of home users have the default links like Control Panel, Games, and Documents on right hand-side of the Start menu , fewer than 2% of our enterprise customers have this experience.
• Most people have removed some items in this part of the Start menu (with Games and Media Center entry points most often removed).
• Enterprise users are launching pinned Start menu apps 68% more often than home users, but the usage of pinned items is still less than 10% of the sessions.

What are we doing with this information?

In general, individual enterprise customers are using Start menus that their administrators have customized. Using this research and our engagement with the enterprise community, we are working on special features that can help address the need for customization in the Start screen. For example, enterprises can remove items like Games and Help & Support from the Start screen. For Windows 8, we support deployment scenarios that include Start screens with a layout of tiles that matches their business group’s needs, allowing for an even greater number of pinned apps to be pre-defined for their users. We also support the managed lockdown of customization of the Start screen so that it is consistent across the corporation. These features have been built especially for our enterprise customers, taking into account the existing functionality that we have provided in the past and the needs that we perceive they will have in the future. And as many know, tech-savvy individuals can use these customizations as well.

Is Start less effective for “at a glance” viewing of my PC?

@mt327000 wrote:

"The Start Screen feels like a mess of icons, having all of the problems with the Start Menu you described and adding some of its own. I keep a very neat and orderly desktop, and can see everything on my computer in a glance in the "All Programs" view introduced in Windows Vista. To me, the Start Screen just doesn't work, nor does it have any advantages over the superior Start Menu."

The comments have been very clear that knowing what’s on your PC and seeing it at a glance is an important aspect of feeling in control. Let’s talk a little bit about how this works in the Start menu and how it compares to the Start screen.

In the Start menu today, when you expand the All Programs flyout, by default you can see a total of 20 apps without scrolling, regardless of how big your monitor is. In one of our studies, we found users launched an average of 57 different apps over the course of several months. And this doesn’t even include the large number of websites that people use day to day (for the purposes of launching and pinning we believe counting websites is important), some of which may evolve into Metro style apps. So you can see how a little window that shows 20 items does not prove scalable in this scenario. The comments have been clear that this scale is routine for those that are reading the blog, and we believe your usage would skew in this direction.

Once apps are installed on the machine, you’ll likely need to scroll All Programs view in order to see all app folders

In addition to the limited real estate, apps in All Programs are buried under folders and subfolders of hierarchy, without any iconography to help you navigate to the right place. To make matters worse, things are often jumping around as you expand and collapse folders looking for the right app, making the experience even less efficient. Some have noted that this limitation is a design regression from the Windows XP Start menu. While technically that is true, we are fundamentally working with a menu, and as such, it is a single column with hierarchy that requires significant dexterity to navigate. The feedback around the scale of the old Windows XP design was resoundingly negative over time and led to the redesign for Vista and Windows 7.

In Windows 8 we assume that there are even more apps (and sites) than the XP/Vista/7 eras and so we needed even more scale. We also wanted to provide an at-a-glance view and a navigation model that requires much less dexterity. By using the full screen, we can now show more apps without the need to scroll or navigate hierarchy. By flattening the hierarchy, we provide a way for you to leverage the iconography of the apps and remove the burden of clicking through folders trying to find an app under its manufacturer’s name. Over time this will also address another common complaint, which is that when renaming, combining, or reorganizing folders (which you might do in order to keep the menu from wrapping) you would lose the ability to uninstall cleanly, and thus subject yourself to a periodic garbage collection of your Start menu to avoid dead links.

As we will talk about later in this post, the dexterity required to navigate a very large menu interface is inconsistent with good user interface design. Even if the items you wish to target are rarely targeted, the whole experience is degraded when constrained to a menu. Some have suggested that using XP-style menus that wrap around the screen, or increasing the size of today’s Start menu would “solve” the issues we are working to solve. Below we will talk about Fitts’ Law and how no increase in size or wrapping will address this. As DPI and monitor sizes increase, it becomes increasingly difficult to zig-zag around the menu to hit narrow buttons. Here is a screen shot submitted via a comment by @Bleipriester, where you can see the mouse “path” required as well as the additional navigation aid of the down/up chevrons. Keep this in mind as we discuss Fitts’ Law below.

@Bleipriester’s evolved Start menu, showing several columns of navigation

Thus as your monitor gets bigger, the Apps screen (an all-apps view of the Start screen) becomes more powerful. Here is how the number of apps that show up in the Apps screen grows across different monitor sizes in our latest builds:

Likely form factor	Size (inches)	Resolution(s)	# Tiles on 1 page of Apps screen	# of Items on 1 page of All Programs
Laptop	12.1	1280x800	36	20
	13	1366x768	40	20
	13.3	1440x900	42	20
Desktop	21.5	1920x1080	80	20
	23	1920x1080	80	20
	27	2560x1440	150	20

Estimated number of apps visible on the Apps screen across different monitors

Your comments have been clear and we agree with many of the design issues you’ve raised. Some of you have mentioned how it’s difficult to find an app when its folder name is no longer available and how completely removing the folder structure has made it difficult to find an app that came in a suite.

To cite @aroush:

“The current metro-list of all apps is not suitable, since that lists everything alphabetically and I don't know the names of all those additional programs."

We are working on addressing this feedback as we speak. Here is our latest design of the Apps screen, which would add back the structure that you’re used to with folders in All Programs today.

You can see here that, as in the Start menu, suites of apps are now organized in groups, instead of in one alphabetical list. This way, if you are looking for something that you know came in your Visual Studio suite, but can’t recall the exact name of the app, it should be much easier for you to find. And your alphabetical list should no longer be cluttered with app tiles that have obscure names because the developer was relying on the folder name to convey the actual name of the executable.

In addition to adding folder structure to this screen and organizing apps within their respective suites, we are also making this view denser. Fitting even more content helps you see what your computer has installed at a glance and decreases the need to scroll. It also decreases the need to navigate a wrapping menu structure or maintain folders or nested folders of programs.

With this design, we improved the scannability of your system, giving you confidence about what is on it at any given time.

Does the new Start support the kind of customization I require to be productive for my work?

@Ed1P wrote:

"While I can see t