[Update 4:40PM PST 10/19/07]
We apologize for not providing updates on this the last few days.  This issue is still ongoing and we are still working to resolve it.  Again, if you are having difficulties with your password in the meantime, please visit the support portal at http://secondlife.com/support.

[Update 5:45 PM PST 10/16/07]
Most of the web team was focused on Friends Online today and did not have a chance to address this issue. However, they will be picking it back up tomorrow and we hope to have a solution for you soon. Thanks for your patience and have a great night.

The password reset tool (http://secondlife.com/password) has been having intermittent failures the last two days. In some cases, the tool will send you the confirmation email and allow you to answer your security question, but then it re-directs you back to the Second Life homepage.

Our web team is aware of this issue and we are working to resolve it as quickly as possible. In the meantime, if you are having difficulties with your password, please visit our support portal at http://secondlife.com/support. We apologize for the inconvenience.

[Resolved 10/3/0 3:45pm PST]
The stipend payment process has now completed and the directory fees have begun processing.  Actually, by the time I wrote that, the directory listing fees were already complete.  Thanks for your patience as we got those processes completed.

[Update 10/3/07 9:15am PST]
Group balances have now paid out, but the stipends have only completed about 70%. As soon as the stipends finish up, the directory fees will begin processing. We’re sorry for the continued delay, but hope everything will wrap up soon. I’ll update when stipends finish and then again when directory fees do.

Good afternoon everyone. It appears that group payouts have not been paid for the past few days, directory listing fees haven’t been charged, and stipends haven’t paid out yet today. We have begun the stipend payout processing now, which will be followed by the directory listing fees. The group balances will begin paying out tomorrow at the regular time. We are working to discover the root cause of these delays and apologize for the inconvenience.

[Resolved 5:30PM PST 08/09/07]
All of the  servers have been moved to different network hardware and will be coming back online as normal.

[Update 5PM PST 08/09/07]
One third of the affected servers have been moved off of the malfunctioning network hardware and should be coming back online shortly. Operations is still working on the other portion and hopes to have all of the regions re-enabled as quickly as possible.

Approximately 200 regions are being taken offline so that their network connections can be moved to different hardware. All region data will be saved. Some regions will come up more quickly than others, but we expect the entire process to be complete in 3-4 hours. We will update with progress as the moves continue.

[Update 1:31PM PST 08/11/07]
Late yesterday we pushed out a network configuration update to the affected hosts which appeared to resolve the issue. Initial data looks promising, but we’ll be continuing to monitor and investigate.

[Update 4:26PM PST 08/10/07]
Following the additional tests, we have concluded that code initially suspected to be at fault was not the root cause and so we will not be making any visible progress on this issue until Monday. To be clear, this does not mean that work on this issue has ceased (in fact, the opposite is true), it merely means that no patch or other fix is likely until next week. Several options in addition to the already written patch are also being researched in parallel, and it is looking like this issue may be the result of a network configuration problem rather than a “bug”. We will update again on Monday or as more information becomes available. Thank you for your patience.

[Update 1:39PM PST 08/10/07 ]
The fix is now being looked at by both development and our QA staff as we get deeper into testing it. Depending on the result of these tests, we may do a limited push of the patch later this afternoon before a full grid-wide deployment. Fortunately, as noted before, this patch will not require a rolling restart so the actual patch deployment shouldn’t cause further interruptions. More updates to come as the testing results come back from QA.

[Update 11:28AM PST 08/10/07]
Our development staff continues to test the potential fix, but does not yet have an ETA for grid-wide implementation. We will continue to post updates as we have them. Thank you for your continued patience.
[Update 9:50AM PST 08/10/07]
Last night, our development staff rolled a potential fix out to a small number of simulators for testing. They will be evaluating the results this morning in consideration of a grid-wide roll-out. Once this evaluation is completed and a decision has been made, we will post further updates.

[Update 6:48PM PST 08/09/07]
The development team is still hard at work on this. We’ll post significant developments or resolution if they occur tonight; otherwise, we’ll check in tomorrow morning.

[Update 5:50PM PST 08/09/07]
Work continues into the evening. Thank you for your patience.

[Update 4:30PM PST 08/09/07]
Our development team continues to work on this issue, but we don’t have any further updates at this time. Thank you for your patience.

[Update 3:30PM PST 08/09/07]
While our developers are still working on a fix for the issue, we do have one small update. It appears that the fix will not require a rolling restart. We will continue to update as we get more information. Thank you.

Just before 2pm our Support staff began hearing from a number Residents about various Secondlife service issues in particular regions. It appears that these issues may have been going on for a few days, but were masked by the network outages we have been working through.

The issue can cause a number of different problems such as: an inability to search or use the map, failure to rez, failures while attaching, L$ balance not being displayed and odd IM behavior. Because there is the possibility of content loss, we encourage you to verify that Search works in the region you are in before attempting to rez any objects.

Our development team has identified the issue and are currently working on a fix.

 [Update 3AM August 8, 2007] Voice Maintenance is complete!  If you do not have your voice dot, please try toggling voice on and off on Edit > Preferences > Voice Chat > Enable Voice Chat.  Be sure to Apply the new setting eacth time.  Otherwise, a relog to a different region can also help. Have a great night/morning/afternoon all and happy chatting!

 [Update 2AM August 8, 2007] The Voice maintenance window has just begun.  I will update again once the servers are confirmed to be down, and once more when they come back up.  We expect to complete the server maintenance before 3AM PST.

Second Life will be performing scheduled maintenance on the Voice System on Wednesday, August 8, 2007 from 2AM until 3AM Pacific (9AM - 10AM GMT). Voice connections will not be available during this time. We will update the blog once the maintenance begins and again when it completes. Thank you.

Good evening.  Earlier this afternoon, several Residents reported general SL performance issues to our Support teams.  After verifying the reports, Support escalated to Operations, and they have been digging into this issue ever since.  It appears that during the grid restart, we maxed out the network connection to the asset server cluster.  This caused a slowdown in downloading and rendering of various textures and objects. We are still looking for the primary cause of this spurt of network traffic so that we can prevent this from occurring in the future.

For now, traffic seems to be stabilizing and returning to normal, but  we will be adding another network switch to increase available bandwidth tomorrow.  Thanks to those who reported this issue to us, and thanks to all of you for your patience.

[Update 5:04pm PST] We have corrected the code responsible for this internally and are currently testing the fix.  There will likely be a rolling restart later this evening, and a blog post to follow with more details on the restart once we have the timing nailed down.

In the past hour or so we have gotten a few reports of objects disappearing from inventory after failing to Rez inworld. The error message accompanying this loss is “Attempt to rez an object failed”. This issue seems to occur following an attempt to rez a no-copy object on a parcel you don’t have permissions on.

Our QA team has reproduced the error and our Developers are currently working on a fix. Depending on the severity of the issue, we may do a rolling restart, so stay tuned for further details. Although, we all know how averse everyone is to Friday updates… In the meantime, please double-check your permissions before trying to rez no-copy objects from your inventory. We will publish more details as we have them. Thank you.

Good morning. Sorry for the late notice, but we are going to begin a rolling restart at about 10:55am (PDT) this morning. This rolling restart is required to roll out some server-side fixes and tweaks. Abbreviated release notes are below. One exciting thing about these release notes is the number that correspond to issues in Public JIRA. Thanks to everyone who has helped identify (and even fix!) issues through this new tool.

These fixes and features require you to be running viewers 1.17.1 or later to take advantage of the updated functionality:
* VWR-796: llStopSound() not working
* SVC-21: Request for making identification of llOwnerSay messages possible
* VWR-1101: Active Gestures > New button doesn’t auto-open gesture * Estate owners can now postpone a rolling restart for an hour

General fixes:
* SVC-242: Copyable objects are not appearing in inventory after being taken, are remaining, invisible, in world
* SVC-52: UTF-8 characters read from notecards are lost
* SVC-248: Viewer crash: Server sends malformed DeRezAck Packet (see VWR-176)
* A few simulator crash fixes

[Update: Pushed the start time back to about 10:55am to allow time for an in-world warning to be sent to all regions.]

[Update: Added links to the issues in JIRA. The restart is now under way. Time to completion is probably 2 hours.]

[Update: The rolling restart is complete as of 1:30pm PDT]

[Update 12:54 PDT]  We have brought additional hardware online to help resolve this issue and logins appear to be back to normal.  Thank you for your patience.

[Update 11:49 PDT] As our loads have increased this morning, we are receiving more reports of outright login failures. These are still related to the same issue and we ask that you continue to wait for the login attempt to time out prior to retrying to prevent exacerbating the issue. We apologize for the inconvenience and are working to resolve this as quickly as possible.

Hello again! This morning we are researching reports of slow logins. It appears that almost all of the login attempts are successful, but they can take a bit to go through (up to 3 - 4 minutes). Please be patient at the login screen and don’t re-try unless your login actually fails with an error message. This will prevent duplicate login requests being sent to the login servers. Our operations team is aware of this issue and is working on it as we speak. More updates here as they become available…

[Update 12:47 PDT] As has been noted in the comments and on the Public JIRA site for this issue, this bug has now been fixed internally.  The fix will be included in tomorrow’s viewer release.  Until then, please use the 1.17.0 version links below if you need to edit an inworld object.  Thank you.

We have received reports through our Public JIRA system that Residents are experiencing intermittent crashes with the 1.17.1 viewer released yesterday. The viewer will sometimes crash when a Resident selects an object and then selects “Edit” from the Pie Menu. You can see more information on the Public JIRA site. Our developers are actively working on correcting this issue and I will be posting updates here.

If you want to return to the previous version of the viewer, you may download the PC version here and the Mac version here.