We’ve just updated the Quality Metrics page, and the numbers show what you already know: April was not a good month for Second Life Grid availability. Our internal outage tracking tool estimates that about 630,000 usage hours were lost to global system failures over the course of the month, which is about 1.9% of the total (up from 0.06% in February and 0.22% in March), and resident surveys clearly indicate great unhappiness coinciding with these failures. (We define lost usage as how much time Residents would have spent logged in but did not, due to Grid failures; it is meant as a global availability metric and does not cover local failures like sim crashes, inventory problems, and the like. See actual[black] vs predicted[blue] concurrency graph excerpt, right.) I’d like to address the causes for this, and what we are doing about it in general terms.

What happened, and Why?
So, on to why April was so tough: virtually every piece of mission-critical Second Life Grid infrastructure failed catastrophically at least once during the month. Here are the biggest sources of downtime; I list these not to shirk responsibility, but to illustrate the near-perfect storm:

• Intra-Grid Network By far, the largest event was the near-total loss of intra-grid network connectivity on April 4/5, caused by a line failure within the backbone of our primary network provider - who delayed, botched, and again delayed the fix, extending what should have been a relatively brief outage for many hours… and then the failure was repeated on a smaller scale on April 25. We’re working with the provider at the executive level to address the risks that allowed this to happen.
• Central Database We encountered a new crash bug in our central database, resulting in a series of four database outages. We have subsequently identified a work-around, and should be able to avoid this particular crash in the future. In general, eliminating this cluster as a scalability bottleneck and failure point is a very high priority for Linden Lab.
• Asset Storage Cluster The asset storage cluster crashed during routine maintenance; this was a repeat of the crashes that afflicted us in December and January. The vendor had previously assured us that this problem was fixed, and so we continue to work with them to investigate the cause and potential solutions.
• Data Center/Transient Data Services Our data center in San Francisco cut power to many of our most critical servers over the course of two days. We had advance notice of this one, but working around the power cuts did cause some disruptions. This was a learning process as well, and we’ve refined our processes for dealing with events like this.

For more information on the systems that make up the Second Life Grid and how, when they fail, they impact the platform, please check the Service Disruptions page. Because component failures are never wholly avoidable, our goal (and the goal of online services everywhere), is to reduce the extent to which the Second Life Grid is affected by the loss of key systems. With a platform this technologically complex, adding the necessary redundancy and failure management is a long process, but we have not been standing still: on average a database crash in April cost about 14,000 lost usage hours, vs 53,000 hours for a similar crash last August (at a time when substantially fewer people were using Second Life). We will continue to reduce the impact (on both logins and in-world functionality) of these crashes, until they cease to be significant, at which point they can come off the Service Disruptions page.

Clearly, though, there is still a great deal of work left to do. Our long term strategy includes specific plans to eliminate the risks associated with all of the failures I listed above. We make progress every day, and continue to hire the best and brightest technical staff at our offices in San Francisco, Mountain View, Seattle, Boston, and Brighton. (See our hiring page.) In the meantime, our service record is not perfect but we are confident that we have identified some key areas to improve and will continue to move forward.

Finally, I’ll mention our new coalesced status reports page, which replaces system status updates on our primary blog. Look there for all Grid-status information, including information about upcoming scheduled outages: http://status.secondlifegrid.net/

Are you following the Second Life Grid Status Reports? If your business, your event, or your time with friends may be impacted by the Grid Status, we want you to have all the information we can get you, as quickly and easily as possible. 

We’ve been posting Status Updates mixed in with a variety of posts on other topics. We realize that many of you don’t want to have to hunt through the blog for needed Status information, so in an effort make it easier for you, we’re following the tech industry standard and moving that information to a Status page. 

Starting today, we’ll be collecting all the Second Life Grid Status Reports onto that one page. Now, you can check for an update without having to scan through all the other blog posts.

You can also subscribe to updates on your mobile phone using Twitter, or with an RSS reader. 

Yep, get your Second Life Grid Status Reports, wherever you go.

Subscribe!

Now you can subscribe to the reports, and make sure they’re sent directly to you.

* Get updates sent to your RSS reader. 

* Subscribe with your mobile phone and you can get Twitter updates while you’re out and about. Go ahead, take that meeting, go back to class, do your shopping, take a walk, get some other work done, and when the next Second Life Grid Status Report comes out, you’ll hear about it.

https://twitter.com/SLGridStatus

RSS: What is it? Status Reports sent Directly to You

• RSS? The BBC offers a good, simple article that goes right to “How do I start…?” 
• If you’d like to get more technical, try Wikipedia 

What is Twitter? Get Status Reports on your Mobile Phone or Computer

• With your Twitter, you can subscribe to updates directly on your mobile phone (or at your computer.) 
• Twitter FAQ
• Twitter Japan, for better service in Japan
• 3 Tips to Get Twitter on Your Cellphone for Free

Quick Links: Second Life Grid Status Reports

• Second Life Grid Status Reports
• Direct link to the new page which collects all Status Reports, so you can easily read the updates.
• http://status.secondlifegrid.net/

Subscribe to the Second Life Grid Status Reports via Twitter

• Want to subscribe to updates with your mobile phone? You can do so with the Twitter feed.
• https://twitter.com/SLGridStatus

Subscribe to the Second Life Grid Status Reports via RSS 

• Grab this RSS feed

[Resolved at 10:50pm Pacific] Our Support Portal is back online!

[Updated at 10:10pm Pacific] Maintenance is in progress. — Frontier

As reported earlier this week, our support portal will be offline for system maintenance tonight, Saturday, 26th April.

Our software supplier has reduced the length of the downtime from six hours to three, from 9:00pm-Midnight PDT.

During that time, the support portal will be unavailable for chat or ticketing services.

Apologies for any inconvenience this may cause to you.

[RESOLVED 22:58] The network situation appears stable, so we’re sounding the All-Clear. Enjoy the rest of your weekend.

[UPDATE 22:18] Ops says the network problem has been fixed, we’ve regained contact with the rest of the world, and things should quickly be returning to normal. We’ll be monitoring the situation, and if no further complications arise, we’ll be giving the all-clear shortly.

There is a major issue causing problems with a number of inworld services, including problems with profiles, teleporting, and maps as well as interfering with logins. Ops is investigating and early indications are that we are being affected by a major networking issue. Please monitor this post for updates.

[UPDATE 25 April 08, 3:40 pm Pacific]

Our payment processor has fixed and tested the process.  All card operations (including updates) are now running within normal time frames.

*****

Due to an issue with our payment processor, we are unable to update your credit card information at this time. Attempting to delete and update your credit card will be indefinitely delayed. You may enter card data but it will not be active in our system for potentially several more hours .

If you have valid credit card or payment information, you can continue to USE it with no problems- this only affects updating card information, or initial card data entry.

Updating Paypal payment information is completely unaffected.

We are working with our payment processor, and credit card updating will work as normal as soon as their issues are resolved.

[UPDATE 1:35pm  Pacific] Logins are open, and the asset system has returned to full functionality. Please be patient when logging in, as the login queue will be processing a high volume of requests for the next few minutes.

[1:02 pm Pacific] Logins have temporarily been restricted to staff-only as Operations addresses a slowdown in the asset system. We’ve also broadcast a request in world for residents who are currently logged in to refrain from manipulating or transferring assets. We’ll have more info ASAP.

[CLOSED 11:07 a.m. --teeple]

All regions affected have been returned to service.  Thanks for your patience!

[UPDATE 10:08 a.m. --teeple]

The vast majority of regions are back online. Operations is still working on a small number of them. We’ll let you know as soon as they’re done.

***

As a result of a recent Rolling Restart, many regions in Second Life are currently down. Our Server Team is working quickly to get them back online as soon as possible. More Updates here as information is received. Thank you for your continued patience.

Our support portal will be offline for six hours of system maintenance this Saturday, 26th April from 6:00pm-Midnight PST. During that time, the support portal will be unavailable for chat or ticketing services.

As originally reported here, our auction pages will be offline for scheduled maintenance this evening from 9 pm until 11pm PST.

We apologise for any inconvenience this essential event may cause.

[Completed 10:41am] Situation with stipends has been corrected.

As anyone who has been here long knows, Second Life is a world that is constantly evolving and changing. Many of those changes are a result of growth. As our population has continued to grow, we are finding it is taking longer to process weekly stipends. Traditionally they have always been paid out on Tuesday. This is not changing, but you may find they do not show up in your account until later in the day on Tuesday or possibly even on Wednesday in some cases.

Nothing has changed in how stipends are distributed and everyone who is entitled to a stipend will receive theirs in due course. That course is just taking a little longer than it used to.

We will update this post when we have word that all stipends for this week have been processed.