Have you traveled between regions and seen pretty yellow-gold messages on your screen letting you know that the region is running a different server version? Welcome to Het Grid, short for Heterogenous Grid.

Check out this weeks KBAotW if you would like to know more About the Heterogenous Grid. This article is packed full of information and will also explain those region notifications you’ve been seeing!

As always, if you have questions about Second Life, and you think the answer should be in the Knowledge Base, please stop by Documentation Office Hours today at 2pm PDT. We love to hear from you! You’ll find our valiant and scholarly Documentation team at our usual pier-shaped office in Beaumont.

Kate Linden

Resolved 5:55 p.m. PDT The database seems to be happy again, consider this the all-clear for transactions. Thank you for your patience.

Update 5:39 p.m. PDT We’re still working on resolving these issues and will continue to keep you updated as the situation progresses. In the mean time we suggest that you avoid making any important transactions until these issues are resolved.

Update 4:40 p.m. PDT We have noticed some problems with teleporting and inventory as well. We’ve investigated the cause of these problems and we’re expecting service to return to normal shortly.

As we have blogged in prior posts, during peak hours we sometimes see a delay in services. We have been working to update our database clusters and are still working to add servers. At the moment some in world services are unavailable including L$ Transactions. Please refrain from conducting transactions at this time until we give an all clear.

Resolved 4:35 p.m. PDT The Issue Tracker and the Wiki are back up and running, thank you for your patience during this outage.

Update 3:15 p.m. PDT Our Issue Tracker and public Wiki remain unavailable. We are in communication with the host company and will keep you updated.

The public JIRA and Wiki are down at the moment. We expect they will be available again in another ten minutes. Updates will be posted here.

[Completed 12:07 p.m. PDT] Our system engineers have completed work on the Central database cluster and services have resumed.

We are about to undergo a 30 minute downtime in order to work on our Central database cluster. All services including transactions, logins and in world services will be affected.

Resolved 5:30 p.m. PDT Logins, transactions and other in world services have resumed. Thank you.

Residents reported interruption of some services which included object rezzing, L$ transaction and login. This was the result of bringing some login servers online before they were ready. Services will resume shortly.

Resolved 3:38 p.m. PDT The support portal is up and running.  We look forward to assisting you.

Update 3:14 p.m. PDT Our support portal is available. Some residents are receiving a 500 error message when attempting reach support. We will have this corrected as soon as possible.

Our support portal is currently unavailable. Residents might receive a 500 error message. We are on top of this and will have the support portal up and running as soon as possible.

[07:54 AM - Resolved] The database upgrade has now been completed. Thank you for your patience whilst this was going on. - Matthew

[07:05 AM - Update] The database upgrade is now under way. - Matthew

As part of a plan to increase the performance and stability of Second Life, we will be upgrading one of our central database systems on Wednesday, April 9th between 7:00 a.m. and 8:00 a.m. As a side affect, the following services will be impacted or disabled:

Second Life Functions:

• Logins
• Teleporting
• L$ Transactions
• Profiles

This upgrade is one more step towards improved performance and reliability of Second Life. We appreciate your patience during this hour.

[Resolved 7:15 p.m. PST] Profiles should be working again!  Please be sure to clear group cache and/or relog for return of services.

Update 6:09 p.m. PST We have identified the cause of profile greyness and inaccessibility. Stand by for working profiles. Once we have completed the fix, you will need to clear group cache and/or relog for return of profile service. We will continue to keep you informed.

Following the work done on Groups and Profiles, residents are reporting grey profiles. This is confirmed and we are now tracking down the cause. Relogging does work in some cases and we will update you here as soon as we have additional information.

[RESOLVED] 5:23 a.m. PST We have been monitoring these issues throughout the night and there would appear to be no further instances at this time.

Update 7:05 p.m. PST Occasional reports of incomplete transactions are still being received.

Update 6:46 p.m. PST Within the last four minutes we have reports of login failure. Our database hiccuped and logins should resume shortly.

Update 5:25 p.m. PST We do still have intermittent reports of login difficulty.

Update 4:22 p.m. PST We are still receiving scattered reports of delayed or incomplete transactions. If after five minutes and/or relog your transaction has not completed, please submit a support ticket for assistance.

As mentioned in a previous post, Operations will be putting out fixes to alleviate the load caused by continued use of our databases. We will be adding additional servers and improving the performance of our database clusters. Until the servers are in place, it is good practice to check the blog for updates, particularly during peak hours of 12:00 pm. - 4:00 p.m.

During these peak hours, we have a few reports of Logins requiring multiple attempts, and scattered reports of delayed or incomplete transactions. If a transaction has timed out or not completed, please wait five minutes for it to complete and/or relog. If the transaction still has not completed, please submit a support ticket so that we can assist you.

[Completed 4:30 PM PST] Operations has concluded changes and restoration of group and profile services. Please clear group cache via the debug/Advanced menu or relog to have group and profile services restored.

Update 12:13 PM PST: We will begin these group changes earlier than originally expected and will commence momentarily.

As part of a plan to increase overall stability of the grid today during peak usage hours, our operations team will make some changes at 1:00pm SLT that will reduce overall database load and create a more reliable experience for everyone. As a side effect of these temporary changes, some group and avatar profile services will not be available.

Specifically:

* Avatar profile information will not be trasmitted to the viewer. This affects both floating and
embedded profile windows.

* General group information (name, charter, etc.) will not display
in floating or group embedded group info windows.

* Groups will not show their member lists.

* Group owners and officers will not be able to eject group members.

* Group proposals will open the UI, but will fail to create.

* About Land will show 0 for traffic.

Please note that all of these effects are temporary and will return to normal behavior when we re-enable these services at approximately 4:30pm SLT. At that time you should either relog or run Client -> Clear Group Cache (a Debug option) in order to refresh group behavior.
We will update this blog post to indicate when these services have actually been disabled, and again when they are again re-enabled.

Over the next week or two, we will be making some changes to the database cluster that we believe will significantly reduce the effects of peak loading that many of you have experienced over the past several weeks. The mitigating measure we’re taking above is something we will only use until those permanent changes are in place.

Thanks for your patience.