Debugging and fixing the BUGCODE_USB_DRIVER (fe) Blue Screen of Death

A remote laptop user has been suffering from occasional blue screens of death with the error BUGCODE_USB_DRIVER. I asked him to email me the mini-dump that had been generated by the last BSOD so that I could analyse it with WinDbg.

Output from analysis of dump file using WinDbg

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************
BUGCODE_USB_DRIVER (fe)
USB Driver bugcheck, first parameter is USB bugcheck code.
Arguments:
Arg1: 0000000000000008, USBBUGCODE_RESERVED_USBHUB
Arg2: 0000000000000006, USBHUB_TRAP_FATAL_TIMEOUT
Arg3: 0000000000000005, TimeoutCode: Timeout_PCE_Suspend_Action3 - PortData->PortSuspendEvent
Arg4: fffffa8007dfcc80, TimeoutContext - PortData

Debugging Details:
------------------
CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0xFE

PROCESS_NAME: System

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff88006830a5c to fffff80002ed8bc0

STACK_TEXT:
fffff880`035d9ad8 fffff880`06830a5c : 00000000`000000fe 00000000`00000008 00000000`00000006 00000000`00000005 : nt!KeBugCheckEx
fffff880`035d9ae0 fffff800`031cfc93 : fffffa80`07df6050 00000000`00000001 ffffffff`dc3a58a0 fffff800`0307e2d8 : usbhub!UsbhHubProcessChangeWorker+0xec
fffff880`035d9b40 fffff800`02ee2261 : fffff800`0307e200 fffff800`031cfc01 fffffa80`036d9000 fffffa80`036d9040 : nt!IopProcessWorkItem+0x23
fffff880`035d9b70 fffff800`0317473a : 24d524c5`24c524c5 fffffa80`036d9040 00000000`00000080 fffffa80`036669e0 : nt!ExpWorkerThread+0x111
fffff880`035d9c00 fffff800`02ec98e6 : fffff880`033d7180 fffffa80`036d9040 fffff880`033e1fc0 54d3e93c`92e2655f : nt!PspSystemThreadStartup+0x5a
fffff880`035d9c40 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KxStartSystemThread+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
usbhub!UsbhHubProcessChangeWorker+ec
fffff880`06830a5c cc int 3

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: usbhub!UsbhHubProcessChangeWorker+ec

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: usbhub

IMAGE_NAME: usbhub.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 52954e12

FAILURE_BUCKET_ID: X64_0xFE_usbhub!UsbhHubProcessChangeWorker+ec

BUCKET_ID: X64_0xFE_usbhub!UsbhHubProcessChangeWorker+ec

The important bit here are the arguments following the Bug Check 0xFE: BUGCODE_USB_DRIVER. These parameters give the exact underlying error associated with the general BUGCODE_USB_DRIVER error. The parameters indicate the failure was due to ‘Timed out waiting for a suspend-port request to complete.’

A highly recommended solution to this issue is to disable USB Selective Suspense in the power settings.

Open the Power Settings from the Control Panel and then click Edit Plan Settings for your current plan.

power_settings_1

Click on Change advanced power settings.

power_settings_2

In the Advanced settings window for Power Options scroll down to the USB settings and expand them to display the USB selective suspend setting. It should be enabled by default. To disable it just click on Enabled and then in the drop down menu that appears change the option to Disabled and then Apply the change and close the window.

How to whitelist a domain in Office 365 Exchange online

We receive automated emails from a domain other than the one we use for staff and some of these emails were getting misidentified as spam and moved to people’s Junk Email folders in Outlook. So we needed to white list the domain so that any emails originating from there would bypass the spam filter.

  1. In the Exchange admin center click on Mail Flow.
    exchange_mail_flow
  2. Next create a new rule by clicking on the + icon and click Bypass spam filtering…
    new_rule
  3. Select on the *Apply this rule if… for The sender… domain is
  4. Add the domain you wish to whitelist plus any additional domains you also wish to whitelist.
  5. Select Stop Processing more rules and then click save.

How do I see who has set up an alert in SharePoint?

Users were supposed to have created an alert so that they received an email about any additions to the Company Announcements page in SharePoint.

We now had a requirement to see if everyone had followed instructions and subscribed to the Company Announcements feed. But how to do that?

It is actually trivially easy to check. Open Site Settings for the relevant page. Then click on User alerts.

SharePoint alerts

From this page you can click on the drop down box to see who has set up an alert for this page.

The redo log is corrupted. If the problem persists, discard the redo log.

Yesterday about an hour before the end of my work day one of our critical servers fell over and was displaying the following message in the vSphere client.

The redo log of VisualSVNServer_1-000001.vmdk is corrupted. If the problem persists, discard the redo log.

The error message refers to a redo log, but this is legacy VMware terminology. VMware have from ESXi 3.1 started to use the term snapshot to mean the same thing but for some reason the error messages still use the old term.

The server was named Subversion and was a VisualSVN Server.

There was a snapshot dated from 15th December 2013 in the Snapshot manager for the Subversion VM so returning to this snapshot would have meant returning to a point several weeks ago and then trying to import the backup of the repository that was made the night of 29th January.

The underlying cause of the corruption cannot be definitively determined but I think was due to the amount of disk activity on the physical disk that constitutes datastore 3_2 on the host server S003-ESXi. This caused the system to fail to write to the log and to create updated delta disks which contain all the changes to the disks since the point of the snapshot.

I believe that if there had not been a snapshot the data corruption probably wouldn’t have happened. I have since educated staff that taking snapshots in vSphere is really not the same as backing up the server and they shouldn’t be doing it on the Subversion server at all.

I resolved the issue with Subversion by carrying out the following steps.

I clicked OK to the error message in the slim hope that the VM could overcome the glitch itself upon a simple reboot.

This didn’t work. So I started the process of backing up the VM by forcing a shutdown of the machine by virtually cutting off the power and then making a copy of the virtual machine folder on the datastore.

Whilst the copy process was going I checked Virtual Machine Logs, vmware-3.log was completely corrupt and the vmware.log was showing some corruption.

The copy process took over an hour as it was 150GB in total size. Mostly due to the two virtual disks the first VisualSVNServer.vmdk which constitutes the C: drive of the server is 40GB and the second VisualSVNServer_1.vmdk which is the E: drive is 100GB.

Having made a copy of everything I attempted to fix the snapshots. I made sure that there was sufficient space on the datastore and then using Snapshot Manager in vSphere created a new snapshot of the Subversion VM.

This operation was successful, so I then tried to commit the changes and to consolidate the disks. This worked for VisualSVNServer.vmdk merging all the changes, but not entirely for VisualSVNServer_1.vmdk, however it did reduce the size of the delta disks significantly meaning that there was likely to be only minimal data lost.

Nothing more could be done through the vSphere client so I then started a process of trying to manually consolidate the following disks into a single disk.
VisualSVNServer_1.vmdk
VisualSVNServer_1-000001.vmdk
VisualSVNServer_1-000002.vmdk

Enabled SSH on the host server s003-esxi.

Using PuTTY I logged into the command line of the host and changed the directory to the relevant directory that contained the virtual machine files for Subversion /vmfs/volumes/Datastore3_2/VisualSVNServer

Then ran the command ls *.vmdk –lrt to display all virtual disk components.

Then starting with the highest number snapshot ran the following command to clone the disk in a way that would merge the delta disks into a copy of the main disk.

vmkfstools –i VisualSVNServer_1-000002.vmdk VisualSVNServer-Recovered_1.vmdk

This process took another hour or so as it was trying to create a 100GB file.

This failed with the following error message displayed:

Failed to clone disk: Bad File descriptor (589833)

Then starting with the next highest number snapshot I ran command to clone the disk without the most recent changes.

vmkfstools –i VisualSVNServer_1-000001.vmdk VisualSVNServer-Recovered_1.vmdk

This process again took about hour as again it was trying to create a 100GB file.

Again this failed with the following error message displayed:
Failed to clone disk: Bad File descriptor (589833)

Abandoned the idea of merging the disks I removed the VM from the inventory in vSphere and then moved all but the following files into a separate folder.
VisualSVNServer.nvram
VisualSVNServer.vmx
VisualSVNServer.vmdk
VisualSVNServer_1.vmdk

I could then recreate the VM from these files. I downloaded the file VisualSVNServer.vmx which is the virtual machine’s configuration file and stores the settings regarding the virtual devices that make up a virtual machine. I edited the file to change all references to VisualSVNServer_1-000002.vmdk to VisualSVNServer_1.vmdk so that the machine could be booted up ignoring the delta disks and any data they might contain.

Added the VM back into the inventory and then booted up the machine. It booted up fine, checked the E: drive and there appeared to be data written to the disk all the way up to the time that the server fell over so it appeared that there was minimal if any data lost.

Thanks to XtraVirt for the necessary steps.

Microsoft renames SkyDrive to OneDrive

Following the threat of legal action from BSkyB for trademark infringement for the use of the name SkyDrive Microsoft came to a settlement whereby they agreed to change the name of the service.

I speculated at the time that Microsoft might have been thinking about rebranding the service anyway as they didn’t seek to fight the case at all. It’s been a while but they have announced that they are changing the name of the service to OneDrive.

I think that it is a very good change particularly the tag line ‘OneDrive for Everything in Your Life’. Because it is true that as the number of devices a person has increases the more necessary a single repository for important files which can be seamlessly accessed from any of the devices becomes.

I’ve already started the process of storing all my photographs online in SkyDrive as a backup for my home PC, but being able to access them on my phone is great and will likely become something that I’ll be wanting to do more often once I become a father this April.

New Spiceworks profile

I use the Spiceworks app as an IT systems management and inventory tool in my company but until recently have only participated in the community very rarely.

Things have changed since they launched their new style of profile page which allows community members to create a ‘portfolio’ of IT projects that they’ve worked on or led.

Spiceworks profile

Eli Etherton believes that they are better than a résumé for IT professionals and I’d be inclined to agree to some extent. For more senior technical roles it is crucial to be able to demonstrate experience with the relevant technologies and it is difficult to condense some IT projects into a bullet point. I can easily sum up one project e.g. Migrated 85 mailboxes into Office 365 Exchange Online.

But setting up a branch office and all that actually entailed is more difficult and the Spiceworks profile allows me to flesh out and illustrate the project.

There is a downside and that is will the recruitment process actually allow me to show off my skills and experience through this new means. Recruitment Agencies and Human Resources departments will favour CVs and LinkedIn profiles because they are the tools that they use and they act as gatekeepers between the IT professional that is seeking a new employee and the IT professional that is seeking a new role.

Free Microsoft Virtualization Training for VMware IT Professionals

Microsoft are really pushing the idea that system administrators that have VMware experience should become bilingual in server virtualisation and get up to speed on Hyper-V too. So following on from the Microsoft Virtualization for VMware Professionals Jump Start, a year or so ago, comes Free Microsoft Virtualization Training for VMware IT Professionals. December 11th from 9am – 12.30pm PST (5pm – 8.30pm GMT)

Get the edge in your technical career! Attend the online Virtualization IT Camp for VMware IT professionals and expand your virtualization skills. Seasoned experts will demonstrate key scenarios and cover equivalent technologies from Microsoft and VMware. Here’s your chance to upgrade your Microsoft Virtualization skills for FREE.

I consider myself already fairly bilingual as I have a Windows Server 2012 Hyper-V host at work with a couple of production servers on it now to go with the 108 virtual machines on our VMware infrastructure. I passed the Windows Server 2008 R2, Server Virtualization exam a couple of years ago when Microsoft was giving away free exam vouchers for it.

Plus I attended the Server Virtualization w/ Windows Server Hyper-V & System Center Jump Start online last month. Just need now to schedule the 74-409 exam whilst my free exam voucher is still valid. The vouchers can still be obtained here (Limited availability)

New virtual server in DMZ not accessible

​I had created a new server as a test environment for a new client of the company and configured it to reside in the DMZ with an external IP address so that people at the client could test the system from their location.

I tested connectivity to this new IP address and the server was connectable and everything seemed fine.

However one of our implementation consultants reported that he wasn’t able to access the server from his location using the IP address that I had provided to him. I tested it again and again it all appeared fine.

I then tried connecting to it from a different network outside of the company and I hit the exact same problem as my colleague had ‘TTL Expired In Transit’. So I then tried a TraceRoute to see if this revealed where the issue might be.

At first glance it appeared okay, traffic was being bounced back correctly from each router along the way. Then I saw the problem, it was because of a configuration error in our ISP’s routers which meant that traffic coming from outside of their network that was destined for the IP address I had assigned to the server was getting routed to a particular couple of routers which were then just bouncing it back and forth between the two of them until the TTL expired.

Converting Windows Server 2012 Standard to Windows Server 2012 Datacenter

It probably won’t come to this as I have now convinced the management to allow me to purchase Vsphere Essentials Plus, but I was curious about whether I could convert my Windows Server 2012 Standard Server to Windows Server 2012 Datacenter without having to do a complete reinstall.

Good news! It is possible and is dead simple to do. Via http://technet.microsoft.com/en-us/library/jj574204.aspx

From an elevated command prompt run the DISM tool and pop your new key in.

DISM /online /Set-Edition:ServerDatacenter /ProductKey:[Datacenter key, e.g. XXXXX-XXXXX-XXXXX-XXXXX-XXXXX] /AcceptEula

CCNA, MCITP and MCSE: Server Infrastructure