:: Deepquest ::

An insight into the problems faced by Microsoft indealing with the Slammer (aka Sapphire) worm is revealed in internal company memos leaked to El Reg.

The email memos from Microsoft security team (18 in all of which we publish only the first [most illuminating] four) reveal a giant corporation struggling to contain the effects of a virulent worm, which knocked many of its Internet services offline.

Those who blame system admins for the spread of the worm might do well to ponder on the fact even Microsoft had not applied its fix which, it’s now clear, was far from easy to apply.

From: Mike Carlson (ITG)
Sent: Saturday, January 25, 2003 8:04 AM
To: EIS Data Center Clients; uDRM Operations Team; Gregory Wood’s Direct
Reports; IR/ID
Cc: HUSK:Corporate Data Center Release Management; ITG DR Command Team; ITG DR Functional Command Team; Randy Coggan; Jim DuBois Direct Reports; Steven Rees

< Microsoft Confidential - Do Not Forward! >

Event Description:

At approx 10:00 pm traffic on the corporate network jumped dramatically, eventually bringing nearly all services to a crawl. The root cause appears at this time to be a virus attacking SQL servers on UDP port 1434. The problem is not confined to MS only, as there are reports of widespread impact at other companies and across the Internet.

Start Date/Time:

Approximately 10:00 PM PST, Friday 1/24/2003.

Impact:

All apps and services are potentially affected and performance is sporadic as best. The network is essentially flooded with traffic, making it difficult to gather details concerning the impact.

Status Update:

Confirmed SQL SP3 does protect the system from the virus, but believe we may have a hot fix that could be remotely applied to SP2 as well. All systems owners should make plans to update their systems asap.

ETA: None.

Action Required:

Owners should be planning to upgrade their systems asap.

Next Update: 0900

From: Mike DeGooyer
Sent: Saturday, January 25, 2003 9:15 AM
To: EIS Data Center Clients; Gregory Wood’s Direct Reports; IR/ID
Cc: Broadband Networking All; Corporate Data Center Release Management; ITG
DR Command Team; ITG DR Functional Command Team; Randy Coggan; Jim DuBois Direct Reports; Steven Rees

Microsoft Confidential

Do Not Forward

Corporate Network Outage: 1-24-03

Update: Network traffic is still affected and inhibiting traffic. Effected resources have been identified and operations is isolating network traffic to apply the fix upon completion.

Next Update: 1000 PST or when implementation steps are ready.

Action Required:

None at this time. Effected resources owners will be contacted.

Description:

At approximately 10:00pm PST traffic on the corporate network increased dramatically due to a virus attack directed at SQL Server. The interruption was not directed at Microsoft Resources.

Event Summary:

8:00am Specific file versions and server impact is being assessed.

7:00am Engineering testing application of hot fix that could be remotely applied to SP2.

6:00am Engineering confirmed SQL SP3 protects systems. Preparing to engage effected server owners.

5:00am Identifying that patch that will fully protect the systems.

4:00am Virus behavior was identified. Working on resolution process.

3:00am SQL Dev and engineering are engaged isolating the issue.

2:00am Engineering engaged to isolate the issue and behavior patterns.

1:16am Notification sent clients.

____________

Mike DeGooyer
EIS Release Management- BGIT

[contact details deleted]

From: Mike DeGooyer
Sent: Saturday, January 25, 2003 11:23 AM
To: Mike DeGooyer; EIS Data Center Clients; Gregory Wood’s Direct Reports;
IR/ID
Cc: Broadband Networking All; Corporate Data Center Release Management; ITG
DR Command Team; ITG DR Functional Command Team; Randy Coggan; Jim DuBois Direct Reports; Steven Rees; uDRM Operations (MSE); Corporate Data Center Release Management

Update: HELP NEEDED: If you have servers that are nonessential, please shut down the MSSQLSERVER service as well as SQL Agent (so SQL doesn’t restart) so that we can eliminate nonessential noise/traffic on the network. Your urgent assistance with this will be very helpful.

SQL Development and Engineering are engaged. Network traffic is still affected and inhibiting traffic. Operations are isolating network traffic to apply the fix upon completion.

Next Update: 1200 PST or when implementation steps are ready.

____________

Mike DeGooyer

EIS Release Management- BGIT

[contact details deleted]

From: Chad Lewis
Sent: Saturday, January 25, 2003 12:35 PM
To: EIS Data Center Clients; Gregory Wood’s Direct Reports; IR/ID; ITG DR
Command Team; ITG DR Functional Command Team
Cc: Broadband Networking All; Corporate Data Center Release Management;
Randy Coggan; Jim DuBois Direct Reports; Steven Rees; uDRM Operations
(MSE); Phil Nguyen (ITG); Mike DeGooyer; Building 11 MCSS Members; TK MCSS
Staff; Jim Pauley; Global Networks Operations Center; Steven Rees; Paul
Olson; Peter Tutak; MSN IA Core

**Microsoft Confidential – Do Not Forward!**

Status Update:

If you have SQL servers that are nonessential, please shut down the MSSQLSERVER service as well as SQL Agent (so SQL does not restart) so that we can eliminate nonessential noise/traffic on the network. Your urgent assistance is required.

Within the next 60 minutes, the 039 SQL patch will be scripted for deployment to all SQL servers outside the data center to reduce traffic volume on the network. For details on 039, visit
[url=http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/bulletin/MS02-039.asp]www.microsoft.com/technet/treeview/default.asp?url=/technet/security/bulletin/MS02-039.asp[/url]
Impact will be a cycling of the SQL service.

We are investigating the best method of addressing the data center space and will communicate during next update.

Next update 1:30pm, or as needed.

Chad Lewis

EIS Release Management

by john.leyden from [url=]http://www.theregister.co.uk/content/56/29073.html[/url]