[Resolved 7:15 p.m. PST] Profiles should be working again!  Please be sure to clear group cache and/or relog for return of services.

Update 6:09 p.m. PST We have identified the cause of profile greyness and inaccessibility. Stand by for working profiles. Once we have completed the fix, you will need to clear group cache and/or relog for return of profile service. We will continue to keep you informed.

Following the work done on Groups and Profiles, residents are reporting grey profiles. This is confirmed and we are now tracking down the cause. Relogging does work in some cases and we will update you here as soon as we have additional information.

[RESOLVED] 5:23 a.m. PST We have been monitoring these issues throughout the night and there would appear to be no further instances at this time.

Update 7:05 p.m. PST Occasional reports of incomplete transactions are still being received.

Update 6:46 p.m. PST Within the last four minutes we have reports of login failure. Our database hiccuped and logins should resume shortly.

Update 5:25 p.m. PST We do still have intermittent reports of login difficulty.

Update 4:22 p.m. PST We are still receiving scattered reports of delayed or incomplete transactions. If after five minutes and/or relog your transaction has not completed, please submit a support ticket for assistance.

As mentioned in a previous post, Operations will be putting out fixes to alleviate the load caused by continued use of our databases. We will be adding additional servers and improving the performance of our database clusters. Until the servers are in place, it is good practice to check the blog for updates, particularly during peak hours of 12:00 pm. - 4:00 p.m.

During these peak hours, we have a few reports of Logins requiring multiple attempts, and scattered reports of delayed or incomplete transactions. If a transaction has timed out or not completed, please wait five minutes for it to complete and/or relog. If the transaction still has not completed, please submit a support ticket so that we can assist you.

[Completed 4:30 PM PST] Operations has concluded changes and restoration of group and profile services. Please clear group cache via the debug/Advanced menu or relog to have group and profile services restored.

Update 12:13 PM PST: We will begin these group changes earlier than originally expected and will commence momentarily.

As part of a plan to increase overall stability of the grid today during peak usage hours, our operations team will make some changes at 1:00pm SLT that will reduce overall database load and create a more reliable experience for everyone. As a side effect of these temporary changes, some group and avatar profile services will not be available.

Specifically:

* Avatar profile information will not be trasmitted to the viewer. This affects both floating and
embedded profile windows.

* General group information (name, charter, etc.) will not display
in floating or group embedded group info windows.

* Groups will not show their member lists.

* Group owners and officers will not be able to eject group members.

* Group proposals will open the UI, but will fail to create.

* About Land will show 0 for traffic.

Please note that all of these effects are temporary and will return to normal behavior when we re-enable these services at approximately 4:30pm SLT. At that time you should either relog or run Client -> Clear Group Cache (a Debug option) in order to refresh group behavior.
We will update this blog post to indicate when these services have actually been disabled, and again when they are again re-enabled.

Over the next week or two, we will be making some changes to the database cluster that we believe will significantly reduce the effects of peak loading that many of you have experienced over the past several weeks. The mitigating measure we’re taking above is something we will only use until those permanent changes are in place.

Thanks for your patience.

Owning land in the Mainland means you are part of a growing community. There can be many residents affecting performance of an individual region. So how can you do your part to contribute to a healthy region and improved performance?

This Knowledge Base Article, Improving Region Performance, provides information for understanding your Statistics Meter. When rezzing an object on your land, check your meter for changes in region statistics. This will help you to see what kind of an effect your objects have on the region overall, and on your neighbors Second Life experience as well. Working together in the mainland can help to ensure a quality Second Life experience.

You can join Jeremy and Jon at Documentation Office Hours this Friday at 2pm; Beaumont, and discuss the wonderful world of Second Life Knowledge.

[Update 16:06 PST] We are considering this issue resolved however the transaction log on the website is still  a little out of date and there are a few regions still being given attention.  We thank you for all of your input and support while we  fixed our Second Life.

[Update 15:21 PST] The support portal is available. If there are any support requests we can assist with, please file a support ticket.

[Update 14:52 PST] The main database is back up and we are still bringing services online. Logins should be working again, as well as Maps, Search, TP Transactions and our Website. The support portal may be slow to load. We will continue to update you as more services are online.

[Update 14:25 PST] Second Life is still down. We expect to have a backup system online shortly.

Our database has crashed and caused in world services to go down. We expect it will be ten minutes minimum before we are up and running again.

[All clear 13:05 PST] It is currently clear to conduct transactions again. If this issue should return we will re-open this post again. Thank you for your patience today.

[Re-opened 12:32 PM PST] We are opening this issue again as there are more reports of L$ transactions timing out. Please do not conduct transactions at this time and wait for an all clear.

[Resolved 9:44 AM PST] Thank you to those residents who alerted us to timed out transactions. We wanted to provide you with this information as quickly as possible. Our database corrected itself and all is clear for conducting transactions.

At this time we are asking that you do not conduct transactions for L$, land or object transfer. We are receiving reports of failed transactions and are looking into the possible cause. I will update here with information and an all clear when this issue is resolved.

[Resolved] Support phone lines are up and running again. We look forward to providing you with telephone support in addition to Live Chat and Ticket submission.

Support phone lines are experiencing temporary disconnection. Support continues to be available through live chat and ticket submission. Updates will be provided here.

We have started to receive reports of regions losing many of their small objects after the region crashed. This was determined to be caused by an Internet Networking hardware problem between our data centers and one of our service providers. We are already in contact with the service provider and we hope to see this resolved quickly.

Should you see massive object loss in one of your regions, please submit a support ticket and provide the name of the region.

Update 4:48 PM PST While we received reports of inability to login, everything seems to be running well.  I’m resolving this issue. If you try to login and are still unable, please let us know in the support portal.

We have several reports of inability to login. If you are unable to log into another region, please watch this post for an update and we will get you in world as soon as possible.

[9:17PM PST --teeple]  A change to the database configuration, and transactions are flowing again.  Thanks for your patience!

Update 8:25PM PST We are still investigating and hope to have more information for you shortly. Please continue to refrain from in world purchases and transfers.

Update 7:28 PM PST Logins may also be affected. If you are having difficulty with login, please attempt to login to a different region.

We have several reports of stale transactions. Do not conduct transactions at this time, which includes transfer of inventory or L$. We are working to determine the cause and will work to remedy this as soon as possible.