[RESOLVED] Temporary Outage affecting 160 Regions
Thursday, March 22nd, 2007 at 3:52 PM by: Pathfinder LindenUPDATE: We’ve completed the work as of now (6:30pm), and all the affected regions are back online.
—
UPDATE: It is taking us a bit longer than expected to resolve this. ETA for getting this rack and all affected regions online and stable is now 7pm. Thanks again for your patience, and we’ll keep you all posted in this blog entry.
—
The rack of class 4 servers that experienced abnormal packet loss yesterday needs to be taken offline temporarily as we update its network connectivity. This is a critical issue, so we need to do it immediately. The affected regions will be back online in about 1 hour
We’ll keep this blog post updated with details as we resolve this issue. Thanks for your patience.
Here’s a list of the affected regions:
Abandoned
Algira
Amella
Amundsen
Anhalt Hill
Apathy
Arcadia 2
Archimedes
Art
Bam
Banda
Bbang
Bellerophon
Berkman
Bikini Atoll
Blue Lagoon
Bothnia
Bradmoor
Brindamoor
Bug Editorial
Cair Paravel
Calzadi Island
Capelli
Carmichael
China
Coniston
Conway 1
Conway 2
Conway 3
Crater
Creamy Dreamz
Dalarna
Davvi
Deltote
Desert Island
Digeut
Disl
Dodam
EarthQuake
Egremont
Enchanted Isle
Endless Despair
Epilais
Eros
Eros 1
Eros 2
Evalua
Evans
Extasia
Fishii
FlyinTails Airfield
FurNation Prime
FurNation Vista
Garage
Gardens Of Enchanted Bliss
Gilum
Hae Hyop
Haean
Hain
Hanja
Hearts Enchanted
Hikuelo
Hirtaria
Huahine
Huchu
Huckleberry
Hudson
Hwaegi
Hyojong
Immidae
Intemporel
Isolation
Istaria Prime
Itaipu
Ivydale
Jodis
JWT
Ketora West
Kitsune Sea
Kiva Island
Kiyoshi
Kovari
Laelia
Landsort
Larsson
Launa Hideaway
Leipzig
Life
Long Meg
Looking Glass
Maayafushi
Maeshil
Matisse
Media
Mediacom Island
Merino
Millions Development
Miranda
Mohrr
Montezuma
Mystical Kingdom
Nani Koki
NanoGunk S
Nissan Sentra
Nuestro Paraiso
Nuku Hiva
Oehler
Peach Island
Pearl
Pennekamp
Pero
Persis
Phoenix Pass
Plush Chi
Plush Genesis
Plush Mu
Plush River II
Pockwock
Pongsonhwa
Quaker
Rage
Raglan Shire
Ramos Island
Romney
Rosario A1
Rosario A3
Saint Clara Island
Saint Croix
Schwanson Schlegel
SeaSide dAlliez
Seaside Resort
Serpentata
Shiot
Shubelik
Sidra
Silvermoon
Sinfull Pleasures
Skouros
Solbitgil
SOUTH I
Soyeonpyeongdo
Spurlock
Suduffco
Svarga
Tahaa Raiatea
Taurus
The Sirens Grotto
Theretra
Therianation
Thunder Ridge
Thurston
Turmus West
Umbriel
UT Austin SL
Victoria
Vixens Isle
VooDooLive
Wagaland
Whitefish
Yeot
March 22nd, 2007 at 4:03 PM
A notification would have been nice, crashing sucks:-(
but as allways you had your reasons for it.
March 22nd, 2007 at 4:08 PM
yay, I hope this helps Vixens Isle, it was getting pretty bad.
March 22nd, 2007 at 4:16 PM
Might wanna take a look at Hundertwasser as well. The sim has been randomly going 1-2 fps for the last week.
March 22nd, 2007 at 4:18 PM
Well, my packet loss on Silvermoon wasn’t abnormal. It was astronomical, but not abnormal. We’re pretty used to crashing every 15 minutes there. I so hope this at least gives us a little fix.. although somehow I feel it’ll be akin to putting a bandaid on an opened artery.
March 22nd, 2007 at 4:21 PM
Ya know… this is the second day in a row I’ve been knocked offline because of this issue.
I’ve been talking to various LL folks for about a month now complaining about the unusual packet loss, but at the VERY least a 30 second warning would have been enough to hit my TP HUD and *Click* *Click* *BOOM* my way out (gratuitous plug intended).
I am glad you are working on a solution guys… but next time… how about a “YO DUDE! Your sim is goin down…” Even a security orb has to give notice…. why don’t LL employees?
/me shakes his head and goes to throw darts at Linden Bears….
March 22nd, 2007 at 4:22 PM
They musy have been bad then cos Class 5’s have been patchy all day
March 22nd, 2007 at 4:29 PM
Well if this fixes packet loss for the regions listed above - maybe some attention can be paid to all the other areas of the grid (most it seems) that have also been experiencing high levels of packet loss and degraded signal for the past week.
March 22nd, 2007 at 4:30 PM
Khisanth Offline
Not on this list and been down for a while!
March 22nd, 2007 at 4:39 PM
Is that list anything like definitive? I’ve been kicked out of a sim that wasn’t on the list, as have a number of my friends; and I can’t seem to log back in either, to _another_ sim that isn’t on the list.
I hope this does resolve some problems. After things being fairly stable, SL’s gotten almost as bad as it was in the Bad Old Days of last November.
March 22nd, 2007 at 4:39 PM
could someone please log me out……………ty…i imagine i,m crashed in a heap on the floor and ppl stepping over me
March 22nd, 2007 at 4:56 PM
@ 8
Yes, that list is definitive. Only those regions are being affected by our update.
If anyone is experiencing issues with a region not on that list, please contact Help Request inworld and notify a Liaison. If you own a private island that is not on that list and is having problems, please contact the Concierge group. That’s the best way to make sure we know about any region having problems. Thanks!
March 22nd, 2007 at 4:57 PM
I agree it would have been nice for a 5 min warning to save all work. We got that with the rolling restarts. Look like 45 min of building wasted.. yet again.
March 22nd, 2007 at 5:00 PM
I’d love to talk to a Liaison. If I could GET inworld I would…
March 22nd, 2007 at 5:02 PM
Another thing folks can do to help alert us of regions not on this list that are having problems right now is to add a comment to this post with the text “[Region Issue]“. That will help us find all region issues in the comments. Also, please indicate how long the region has been down. Thanks.
March 22nd, 2007 at 5:07 PM
Pathfinder,
What is the status of regions on this list comming back up?
Your original post said “about 1 hour” and it’s past the 2 hour mark now and my home region (which IS on the list) has not come back up yet.
Thanks for your time in responding!
March 22nd, 2007 at 5:18 PM
@ 13
I posted an update on the main blog entry.
March 22nd, 2007 at 5:40 PM
Still having issues TPing to many regions…
March 22nd, 2007 at 5:57 PM
The Freinds list is broken again.
March 22nd, 2007 at 6:13 PM
@ Pathfinder.
“Yes, that list is definitive. Only those regions are being affected by our update.”
Are you so certain? How can you be so sure?
You are dealing with a very complex interconnected massive network of servers, Taking down one rack of servers can and Will effect other servers/regions in the net work.
That list you provided only lists regions you deliberately and manually taken down.
I have have been following the posts on both yesterdays and todays outage. As you can see by the posts Other regions/servers are being effected by the planned outage. But by what extent is currently unknown.
Of course to be fair you guys had know way of knowing this could happen.
PS good work on staying on top of the server issues.
March 22nd, 2007 at 6:14 PM
not comment
March 22nd, 2007 at 6:19 PM
I just saw an over 45 second frame time, due to “Images Time” — I think it’s fair to say at least some other sims are being affected by SOME problem, if not this one.
http://www.secondskies.com/img/argh.jpg
[Taken from inside Dogfight Atoll.]
March 22nd, 2007 at 6:22 PM
REGION ISSUE
Plush Zeta has been suffering severe packet loss and low framerates for days now. Please look at it and check it out, would be greatly appreciated.
March 22nd, 2007 at 6:43 PM
Thanks, this update finally did something good and didnt revert bugs again like previous ones
March 22nd, 2007 at 6:43 PM
One of the listed sims, namely Pockwock, has been experiencing severe lag at random. I don’t know if this restart fixes the lag issues in the sims or not but even the Sim FPS and Physics FPS have gone under 10 with no clear indication of what could be causing it (Nothing new has been rezed, in fact several things were taking out/offline to aleviate the lag with no effect). Can a linden please go look at pockwock and find out what the heck is going on?
March 22nd, 2007 at 6:50 PM
Region Issue
Just suffered a complete loss of FPS at Pinemont Mainland region, which resulted in a Loss of connection to the regions server, I had to reloge in order to reastablish the connection and restore FPS.
March 22nd, 2007 at 7:48 PM
a rack…. 160 regions…. are you series that 1 rack is running 160 regions? not realyl going to fit more than 32 machines comfortably in one rack… 32 machines for 160ish regions and you wonder why there is lag? …..
March 22nd, 2007 at 8:18 PM
@23 well they’re likely all 1U blades. Also a Class 4 holds 4 normal sims, and up to 16 ‘void’ sims (4 void sims equals 1 normal sim). I’ve seen racks that can hold 40 1U blades (40 x 4 = 160).
March 22nd, 2007 at 8:45 PM
I hate to sound ungrateful for the hard work thats going into this, but what about the lost revenue that this 2 day fiasco has caused? We should have some anticipation of credit or renumeration IMO.
March 22nd, 2007 at 9:48 PM
Stroker, if you want stable platform for a busness, don’t use SL. As its been proven it’s not overly stable. Don’t hold your breath.
that said, If the Lindens could tell when a crash was going to occur, then they woudln’t be crashes would they? they’d be Planend outages..
March 22nd, 2007 at 10:47 PM
Stroker, to be honest, I’ve tried to get to your shop twice in the last two days - both while the sims were down. But you know what? I’ll be back, because I know the quality of what you sell. Just waiting for my shopping partner to come back online :> I really doubt you are going to lose sales because of this.
Granted, this logic doesn’t apply to every little shop in the affected regions. But what Angelita said is true - SL is a great toy, a remarkable achievement that, as a programmer, I’m stunned succeeds as often as it does - and it’s not something that’s mature enough to build a business model on. You spins the wheel, you takes your chances.
Signed, a Very Happy SexGen customer
March 22nd, 2007 at 11:07 PM
[Region Issue]
Khisanth off line for hours!!!!!
March 22nd, 2007 at 11:17 PM
Stroker Says: I hate to sound ungrateful for the hard work thats going into this, but what about the lost revenue that this 2 day fiasco has caused? We should have some anticipation of credit or renumeration IMO.
Perhaps only if all your competition in affected areas are also paid the same amount?
March 23rd, 2007 at 12:34 AM
[...] A rack of 160 simulators was taken offline for an unexplained network upgrade, from 4pm to 6:30pm. We don’t know if the matter was cabling, network cards, router, switches or other. Residents report generally improved performance in those sims subsequently. [...]
March 23rd, 2007 at 12:37 AM
I wonder if this means we are done with this work over all or done with this part of the work? after the first outtage of these regions this week the work was “completed”.
Althought after it was “complete” I still saw the same packet-loss in my region.
So was the first session the investigation and this second session the resolution of the problem? Or, should we expect further sessions of this rack of regions being taken offline?
Would be nice to have some clearer statement about this situation please.
best regards,
LLeaf
March 23rd, 2007 at 12:57 AM
any idea why everything is greying out?
was fine this morning til i crashed , now everything is grey.
March 23rd, 2007 at 1:05 AM
Sorry guys whatever you did today you screwed the grid up again bad. everything is back to grey avatars in all sims and object creation and modification is all screwey and random movement again.
You did NOT fix the issues.
You did however succeed in making it all worse again.
Start over again tomorrow?
Hire people that know what thwey are doing?
Oh wait you have them
Spend a few mill on that real test grid maybe? yes thats what you ahve to do.
Sorry execs. Time for you to give it all back, sell those fancy houses and cars, and pour that VENTURE CAPITAL INTO THE VENTURE.
March 23rd, 2007 at 1:20 AM
oh yea. helpful steps to reproduce:
log into SL
go anywhere
observe massive lag and nothing rezzes. everyone is gray. grid wide.
create an object
set any properties
watch them randomly revert and change before your eyes.
Any sim
grid wide
March 23rd, 2007 at 1:26 AM
whats this about 2 days?
how about since the change to the new backbone messaging system and move to a colo model?
sl has been wrecked since then.
And more specifically we are going on a month of instability and inability to create complex content.
The time and revenue is lost forever and it cannot be made up.
But since free accounts are allowed there isnt much recourse.
the day SL moves to a paid access model they are accountable.
So free accounts will remain forever.
As will sl continue to degrade since there is no real way for them to test anything. the test grid is a joke.
as for the current grey grid debacle, there are 18,756 online. so guess what? it has nothing to do with the free accounts or campers or any of the stuff the whiners usually whine about.
this will be something to do with a failing poor database architecture, messaging architecture, or network architecture.
I dont see any job postings for meta or system architects at linden labs. thats what they need lol.
March 23rd, 2007 at 4:52 AM
lol like the third set of regions to go down due to power outage in a month.
We’ve had months and months of lag lag lag crash fail lag lag lag crash fail lag lag lag crash fail.
It’ll be amusing to watch LL fail when they get it right and everyone has got fed up and left by then.
March 23rd, 2007 at 5:07 AM
Yay - grey avatars everywhere again.
And oh noooooo - return of the Missing Image.
And I dont know what happened during the last 2 days but MANY of the scripts on my region were left in a not-running state when the work was “completed”.
Thanks for that
March 23rd, 2007 at 6:01 AM
Has nothing to do about poor backbone. What viewer are you using? Firstl Look? Normal?…..
I know First look is having all sort of issues, thats because the codes are just not working as steve and company planned. We just need to keep pumping and digging swearing etc until the client(s) which ever you refering to are fixed correctly.
March 23rd, 2007 at 6:03 AM
@36 well missing textures ( i been seeing this for 3 straight weeks when i thought they fixed it then., grey are oh so oh common now! Slurs don`t work for me either.
March 23rd, 2007 at 6:09 AM
The soul of the Grid.
@all the annoyed people:
I was p***** off too - but we all you keep in mind, that this is still all a huge experiment and somehow a wonder how FAST so MANY PEOPLE created an account in SL.
So it’s easy to blame LL, but try to see them as the “victims” of their own success.
The alternatives: shutting down the account creation or just giving “paying” accounts access in peak times?
That would create a completely new debate - but would not cure in the end these immense challenges LL is facing.
I recommend to all the people: listen to the soul of the grid.
There are signs, when it’s getting instable. If you can feel it, you know when it’s time to stop working.
There is no other cluse then: be patient, watch the blog, or feed your pet.
As far as I can say: they are working like crazy… doing their best.
To me it’s like this: You can’t operate on a lung with 10.000 surgeons at the same time… sometimes you not solve a technical problem by hiring more people and just buying more server power.
Bless you,
Mark
March 23rd, 2007 at 6:11 AM
In heavey traffic and or population island/sims the grey is occuring way way too much……….STEVE These are class 5 Islands! what the hech is this? They have a ex 2 giga of ram so( wasnt that suppose to have been the answer to the lag and slow texture loading? Shakeshead. All In All what you expect these day? People don`t start screaming about freebie account.BECAUSE LLABS is has said they are here to stay. So we need to deal with it like it or not. Bitch all you want, Or better yet don`t let them in on your island……..Island owners knows freebie acount rally does help boost traffic/population to a island so………….Throw a money ball out well OF COURSE they come running what you want? You want population in your island right? Island owners knows this so what the hech are you bashing freebie account then? I dislike freebie account myselve. But atleast I have some logic when it goes to understand and then blaming llabs ( when llabs themselves have said they will not remove freebie account.
March 23rd, 2007 at 6:51 AM
#39, you’re kidding right?
The soul of the Grid.
@all the annoyed people:
I was p***** off too - but we all you keep in mind, that this is still all a huge experiment and somehow a wonder how FAST so MANY PEOPLE created an account in SL.
–When you pay for an experiment, (anything really), you get pissed when it’s botched
So it’s easy to blame LL, but try to see them as the “victims” of their own success.
–Don’t forget the other victims the people paying for “their success”.
The alternatives: shutting down the account creation or just giving “paying” accounts access in peak times?
–the second idea was promised and prompty forgotten
That would create a completely new debate - but would not cure in the end these immense challenges LL is facing.
–the promise was overall well accepted in the blog as far as I saw
I recommend to all the people: listen to the soul of the grid.
–you’re making some sort of metaphor here, but it goes nowhere.
There are signs, when it’s getting instable. If you can feel it, you know when it’s time to stop working.
–ah, this must be what you mean by soul. It doesn’t mean the same thing to me. Maybe “health” would have made some sense. Being forced to stop working/selling/playing/using at random times is cause for complaint.
There is no other cluse then: be patient, watch the blog, or feed your pet.
–if people didn’t complain, quit, a major company would be slapped with class action suits for each of these outages and learn that they either have to stop messing around or make amends.
As far as I can say: they are working like crazy… doing their best.
–tinkering might be a better word here
To me it’s like this: You can’t operate on a lung with 10.000 surgeons at the same time… sometimes you not solve a technical problem by hiring more people and just buying more server power.
–it’s not like that.
March 23rd, 2007 at 7:33 AM
@41:
Man - you seem to be really disapointed.
Question: Do you live off doing business in SL?
IMHO: That’s like living in SF and complaining about the risk of a earthquake.
Please don’t take this personal - I just bought your mood mixer and maybe even some more
March 23rd, 2007 at 7:56 AM
A lot worse isnt even the right words XD we are some that cant move, cant walk, cant tp, cant fly, cant sit. More easily said, we can do absolutely nothing other than chatting on IM
March 23rd, 2007 at 9:37 AM
But it truly bugs me when people say “this is an experiment” as tho that accounts for poor corporate communications and by some unfathomable logic makes everything ok.
NOWHERE on the web pages hyping Second Life to potential users does it say “this is an experiment so dont expect it to be reliable or well communicated”.
We buy into the “own virtual land” - “buy & sell virtual money” - “create virtual worlds” - etc etc.
And having paid for your promise, you get hit on the nose daily by the harsh reality that its not living up to its advertising.
(recently I might add - 1 year ago things were much smoother)
Another argument that gets my hackles up is when people say “If you can do better then go do it and stop complaining”.
Wth?
Sooo, if u r not happy with a service in RL, like a builder or plumber etc, you decide to be a builder or plumber and dont complain to them? that’s fairly odd I would say.
So if you dont like the process of complaining to try to get something done. Then dont read the blog replies. Pretty simple.
Because Torley, Pathfinder, Blue, Steve and so on do actually respond to some of this.
Silence and only commenting on successes is head-in-the-sand behaviour that probably leads to even more difficulties on the SL grid & viewer.
The new Issues Tracker is a great step forward and allows for issues (complaints) to be managed more reliably.
Where have the town-hall meetings gone?
Where are the communications that we used to get from Philip and the upper management? Not even proxied communications are made from them via a company communication specialist.
So please Linden Labs - spell out the road map and troubles you have been having.
For one its actually quite interesting to a large percentage of your audience - and most importantly - it will eash peoples tensions and concerns.
best regards,
LLeaf
March 23rd, 2007 at 9:44 AM
Ketora is CURRENTLY experiencing a ridiculous amount of lag, and is a class 4 server that was not on this list. Is something new going on?
March 23rd, 2007 at 1:37 PM
THANK YOU !! for doing something, finally!
March 23rd, 2007 at 6:55 PM
45 Web Page…………….LLabs really does not care about us anymore and what we want. But its instead the money players the care more about. If the First Look CLient is any reason to worry ( with its offen bugged and missed coded releases ) we are in a even longer deep darkness where there is no end.
March 24th, 2007 at 2:18 PM
well i now know why people sign up and are gone 30 days later there are more stable platforms to work and play in and they cost less to
January 28th, 2008 at 8:45 PM
vista windows orb pearl