[UPDATE 1:35pm  Pacific] Logins are open, and the asset system has returned to full functionality. Please be patient when logging in, as the login queue will be processing a high volume of requests for the next few minutes.

[1:02 pm Pacific] Logins have temporarily been restricted to staff-only as Operations addresses a slowdown in the asset system. We’ve also broadcast a request in world for residents who are currently logged in to refrain from manipulating or transferring assets. We’ll have more info ASAP.

[UPDATE 24 Apr, 3:40 p.m. Pacific] The billing process is caught up.  The accounting group and their associated dev team will be monitoring this weekend’s billing run for any sign of trouble.

[UPDATE 22 Apr, 4:55 p.m. Pacific] Last night’s run cleared many accounts, but not all of them. Another catch-up run is scheduled for tonight, and we’ll post an update tomorrow.

Some of you may have noticed that you haven’t been billed yet for this cycle. The biller hasn’t run properly, so collecting some payments has been delayed. We’re planning a special billing run tonight as the first step in catching all accounts up to date.

Don’t worry, your payments aren’t late, and it won’t affect account status or land ownership. Because this is running a few days late, next month’s billing might seem to come up sooner than usual, but it will run on the usual day.

Once we have more details we’ll give you an update. Thanks!

At the top of the hour, I-World Island debuts with a full twenty-four hour showcase of musical talent.  

Here’s the lineup (PST):

• 12 am Mikelec Criss
• 1 am Jackdog Snook
• 2 am Cromwell Hoobinoo
• 3 am Dann Numbers
• 4 am Allister Westland
• 5 am Phemie ALcott
• 6 am Lacoppia
• 7 am Spaceman Opus
• 8 am Erin68 Frog & satin Galli
• 9 am Rich desoto
• 10 am POL Arida
• 11 am Jean Munro
• 12 pm SHamanes
• 1 pm Kaklick Martin
• 2 pm SRV4U Conacher
• 3 pm Virtual Live Band
• 4 pm Jaycatt & Frogg Marlow
• 5 pm Friendly Fire
• 6 pm Kyle Beltram
• 7 pm Zed Essex
• 8 pm Forsythe Whitfield
• 9 pm Kelvinblue Oh
• 10 pm Tallguy Kidd
• 11 pm Von Johin

We originally announced a Wednesday morning (Pacific) outage for the support portal.  That outage has been canceled.  All necessary work was completed during the first phase of maintenance.

[CLOSED 9:45 a.m. Pacific] The power cycling is complete.  All voice and live chat channels are now running at normal capacity.

*****

A minor hardware glitch has prompted us to shut down the power to one of our support teams for about ten or fifteen minutes. This is causing reduced access to voice and live chat for Billing and Outworld support; Concierge support is unaffected. This should take about fifteen or twenty minutes to address at most.

[Edit: in World Support channels are also unaffected.]

[UPDATE 15 Apr, 11:22 a.m. Pacific] The Wednesday maintenance for the support portal is canceled.  All the necessary work was completed during the Sunday night maintenance.

[RESOLVED 22:44] The service disruptions caused by tonight’s maintenance should now be taken care of. Thank you for your patience and understanding. - Twilight

[UPDATE 14 APR 22:27] The maintenance is causing some disruption of services for some residents, including logging in, teleporting and transactions. Please refrain from making any transactions until further notice. Please monitor this post for further updates as they occur. - Twilight

We have one maintenance event scheduled for each day between this coming Sunday and Wednesday, inclusive.

The maintenance is of two types, and each event will briefly impact resident experience in one of two ways.

Sunday 13 Apr, 9pm - 10pm Pacific: Our support portal will be off line for one hour for Phase I of a two-step vendor maintenance event. During that time, the support portal will be unavailable for chat or ticketing services.

Monday 14 Apr, 10pm - 1 am (Tuesday) Pacific: Phase I of an essential two-step upgrade to the breaker system at our San Francisco datacenter facility will require a shutdown of a number of our server racks.

Tuesday 15 Apr, 10pm - 1 am (Wednesday) Pacific: Phase II of the breaker maintenance at our San Francisco facility will again require a shutdown of some of our server racks.

Wednesday, 16 Apr, 7 am - 9am Pacific: The final phase of the support portal maintenance. The support portal will again be unavailable during the maintenance window.

Operations is already at work minimizing impact of the Monday and Tuesday maintenance by rerouting essential backbone services. However, we project that a total of approximately 1400 regions will experience three hours of downtime on one or the other of those nights.

Due to the way regions are dynamically assigned, we cannot predict in advance which regions will be affected, but as we get closer to the start times, we’ll publish any particulars that our own Ops team can control or pre-schedule to minimize impact on resident experience.

[05:27 a.m. Pacific - Resolved] Our Live Chat lines are open again. - Lotte

[4:15 a.m. Pacific] The update is there isn’t an update. We are still working towards a resolution with our support system vendor and look forward to restoring Live Chat as quickly as possible. Thanks for your continued patience. — Frontier

[1:54 a.m. Pacific] We’re still working with the support system vendor to restore Live Chat to full service. We’ll keep you posted.

[12:52 a.m. Pacific] The Live Chat feature isn’t connecting properly to Customer Service representatives. We expect to have it back in service shortly.

[3:04 PM Pacific] All services have been restored. If you find yourself without group lookups or any of the other functionality in the second list below, please either clear your group cache (instructions are toward the bottom of this post) or relog to obtain full services.  Thanks!

[1:20 PM Pacific] Operations is re-enabling the first list of essential In World and Web services below.

The second list, part of our short-term mitigation strategy, will be returning to full functionality over the next 60 to 90 minutes. If that changes, we’ll let you know promptly.

Please be patient when logging in over the next few minutes. As is the case after any full restart, the login queues will be congested. Frustrating though this has been, please try to avoid the temptation to cancel and restart your login, and just ride the process out to keep your place in the queue.

[12:50 PM Pacific] In addition to the reduction of services outlined below, Operations will be immediately disabling the following services for at least 30 minutes:

• Logins
• Lindex
• In-world transactions
• Portions of Website functionality, including access to account pages and the support portal.

We’ll keep you updated every 30 minutes at most on this, and announce a return to service ASAP.

In order to increase overall stability of the grid today during peak usage hours, our operations team has disabled a set of in world functions to reduce overall database load and create a more reliable experience for everyone. As a side effect of these temporary changes, some group and avatar profile services will not be available.

Specifically:

• Avatar profile information will not be transmitted to the viewer. This affects both floating and embedded profile windows.
• General group information (name, charter, etc.) will not display
  in floating or group embedded group info windows.
• Groups will not show their member lists.
• Group owners and officers will not be able to eject group members.
• Group proposals will open the UI, but will fail to create.
• About Land will show 0 for traffic. (Note: This is temporary, and will not cause the loss of traffic captured throughout the time the total is unavailable to display.)

Please note that all of these effects are temporary and will return to normal behavior when we re-enable these services later today. At that time you should either relog or run Client -> Clear Group Cache (a Debug option) in order to refresh group behavior.

We will update this blog post to indicate when these services have actually been disabled, and again when they are again re-enabled.

As previously blogged, these steps are part of a temporary mitigation strategy until additional hardware can be installed, along with other scaling refinements planned for the very near future.

Thanks for your patience.

[10:52 PM Pacific --teeple]

Operations has just addressed an unexpected performance hit on the central database cluster.  For a very few minutes, logins were slow or stalled, and various in world functions such as land and L$ transfers and map/search lookups were inhibited for many residents.

We were just finishing a blog post to report the service disruption when Ops found and corrected the root cause.

In addition,  Ops has corrected a misconfiguration one one of our two new firewalls which was causing timeouts on approximately 50% of attempts to hit our homepage for the past few hours, as well as inhibiting some percentage of inworld LindeX orders and land transfers.

While inconvenient and frustrating, we believe that all stalled transactions resulting from the two gremlins this evening have reverted harmlessly…that is, that no residents have suffered incomplete transactions.  If you have experienced a loss this evening, please contact us via the support portal.

Thank you for your patience, and we apologize for the inconvenience.

[Completed 4:30 p.m. PST - Kate] Operations has concluded changes and restoration of group and profile services. Please clear group cache via the debug/Advanced menu or relog to have group and profile services restored.

As part of a plan to increase overall stability of the grid today during peak usage hours, our operations team will make some changes at approximately 1:00pm Pacific which will reduce overall database load and create a more reliable experience for everyone. As a side effect of these temporary changes, some group and avatar profile services will not be available.

Specifically:

• Avatar profile information will not be transmitted to the viewer. This affects both floating and embedded profile windows.
• General group information (name, charter, etc.) will not display
  in floating or group embedded group info windows.
• Groups will not show their member lists.
• Group owners and officers will not be able to eject group members.
• Group proposals will open the UI, but will fail to create.
• About Land will show 0 for traffic. (Please note: this is temporary, and impacts only the display of traffic, not the recording of it.)

Please note that all of these effects are temporary and will return to normal behavior when we re-enable these services later today.

We will update this blog post to indicate when these services are re-enabled.

Over the next week or two, we will be making some changes to the database cluster that we believe will significantly reduce the effects of peak loading that many of you have experienced over the past several weeks. The mitigating measure we’re taking above is something we will only use until those permanent changes are in place.

Thanks for your patience as we work to improve the Second Life experience..