[DONE] Rolling Restart for 1.21 Server Deploy Wed/Thu/Fri
Saturday, April 26th, 2008 at 5:10 AM by: Joshua Linden[Updated Saturday @ 09:10am] The rolling restart of the rest of the grid is now complete.
[Updated Friday @ 8:39am] The rolling restart to half of the grid is now complete but for 7 hosts that needed to be manually updated; those will be completed within a few minutes. The rest of the grid will be updated tomorrow morning.
[Updated Thursday @ 7:10pm] We are beginning have completed the deploy of 1.21 to 3 racks (632 regions). Here is a list of regions that as of now are on version 1.21.0.85745.
[Updated Thursday at 12:47pm] We will shortly be deploying have deployed 1.21 to 1 rack (about 170 regions) again. If all goes well, we will continue with the tenative timeline listed in the Wednesday at 8:10pm update below.
[Update Wednesday @ 9:15pm] A slight and subtle wrinkle during the deploy left some object-to-object emails non-functional. The responsible systems have gotten a stern talking to, and this service should be operational again.
[Update Wednesday @ 8:10pm] Another bug was found after we rolled out to one rack. That bug has been found and fixed. We will evaluate exactly what we’re going to do with this deploy after testing tomorrow, but it will likely shift the timeline forward by one day. Meanwhile, we are rolling back the 170 regions that had previously received a 1.21 deploy so that for all simulators are once again running on version 1.20.1 of the server code.
The central updates to 1.21 are complete and things seem “nominal” at the moment, but of course we’ll be watching closely.
- Wednesday 4/23 @ 11am - deploy to 1 rack [DONE] [REVERTED]
- Wednesday 4/23 - update central systems throughout the day [COMPLETE]
- Thursday 4/23 @ 6pm - deploy to 3 racks [COMPLETE]
- Friday 4/25 @ 5am-11am - deploy to half of remaining servers
- Saturday 4/26 @ 5am-11am - deploy to remaining servers
[Update Wednesday @ 10:25am]
The bug in the 1.21 Server code identified last night during an initial rollout to 1 rack has been found, fixed, and verified. We’d planning to proceed with the rollout to avoid delaying the code update another week. On the table for today are the central services updates and limited rolling restarts.
What’s Changed in 1.21 Server
The most notable fixes will be physics-related, and have been in testing in the Beta Preview for several days. No new viewer is required.
- Changes to simulator startup to alleviate database load
- Physics-related fixes that have been in testing in Beta Preview - see the full list in this post
- Fix for SVC-245: Math error on derez
- Fix for SVC-1896: Able to make anyone’s objects shout error messages
Read on for more information…
More Details
A “rack” is a physical set of about 40 sim hosts, so about 160 regions, give or take spares. This is also a handy sized unit for initial rollouts. We’ve started doing restarts spread across several days to catch any configuration or scaling issues before they affect the whole service, and also because the service is now so large (10 times as many hosts as when I started, I believe) that we need to do it in pieces
During the central system updates we expect brief disruptions of some services (less than 2 minutes). For example, it may be necessary to re-join group chats, the reported “residents online” numbers may drop, and logins may not function briefly as the processes responsible for those services are stopped/started. These activities are partitioned by agent and group - for example, a particular resident may not be able to log in while one of the hosts is restarted, while other residents are able to log in. This usually takes less than a minute for each of 16 hosts.
1.21 Rollout History
The 1.21 Server deploy was initially scheduled for April 16th/17th. During the rollout, some problems were encountered which caused us to roll back, review the code, make some fixes, and proceed cautiously. In detail:
- a component swap intended reduce the disruption caused by central updates exhibited poor performance in production; this was reverted in favor of a different architectural change already planned, but waiting for a future update
- a data migration intended to alleviate database load during login and other actions turned out to have a subtle bug that required reversion; after further review, the updated code is going out in a multi-phase approach starting with 1.21.
- during an initial “3-rack roll“, several services on the targeted hosts were not started correctly; further investigation could not determine if this was due to a momentary network glitch, database hiccup, problem with the deploy tools, or a problem with the code that was not caught during testing. Subsequent testing was unable to reproduce the problem. Since this is easily detected and recovered from, we’re proceeding cautiously. So far, no issues have been seen.
[Initial Post Details from April 22nd]
We’re ready to initiate the update of the Second Life servers to the 1.21 version of the code.
The deploy is going to happen in several phases:
- Tuesday 4/22 @ 1pm - deploy to 1 rack - [DONE]
- Tuesday 4/22 @ 5pm 7pm - deploy to 3 racks
(delayed a bit due to a wrinkle discovered during the previous step) - Wednesday 4/23 @ 6am - deploy to 10 racks
- Wednesday 4/23 - update central systems throughout the day
- Thursday 4/24 @ 5am-11am - deploy to half of remaining servers
- Friday 4/25 @ 5am-11am - deploy to remaining servers
Should problems be encountered with the 1.21 rollout, we will likely proceed with deploying a subset of the changes focused on physics-related fixes, as we did with last week’s 1.20 patch rollout.
[Update Tuesday April 22nd @ 7:55pm]
After an initial re-rollout to 1 rack, reports came in of attachment failures. The rack currently is being reverted to the previous (1.20.1) simulator version. After some quick tests, we believe we’ve narrowed down the changes responsible (tests for rez permission appear to be checking a remote parcel incorrectly), but a fix is unlikely until tomorrow at the soonest. The issue also affects the “backup plan” for a smaller patch deploy mentioned at the end of the post (which hints at the source of the bug).
[Sorry for making updates at both the top and bottom of this post; I want it to remain understandable for residents who are reading it for the first time, yet retain the history of the post for later comments to remain sensible. -- Joshua Linden]
April 22nd, 2008 at 2:41 PM
Great! Can’t wait to try it
April 22nd, 2008 at 2:46 PM
Oh joy.. oh.. btw.. how about fixing the current problems of no tp/no profiles, etc?? That would be peachy
April 22nd, 2008 at 2:47 PM
Thanks! =)
April 22nd, 2008 at 2:49 PM
Thanks for the in-depth info - very much appreciated.
April 22nd, 2008 at 2:52 PM
Please don’t feed the Trolls.
April 22nd, 2008 at 3:11 PM
Uhm… Second Life is offline?
Second Life Service Status
Second Life Offline for Database Maintenance or Update
In our continuing effort to improve your Second Life experience, we have taken Second Life temporarily offline for maintenance. This may also effect portions of our website. We are working to re-open our doors as quickly as possible and we appreciate your continued patience. For more details and further information on current system status, please visit our official blog at blog.secondlife.com.
April 22nd, 2008 at 3:15 PM
Great blog entry about SL being offline.
Unfortunately, its written in invisible ink.
April 22nd, 2008 at 3:21 PM
Hey this sounds like some good fixes, so that explains why I couldn’t log in and it kept not responding on me. Hehe, silly me. Going to have to wait till it’s done.
April 22nd, 2008 at 3:25 PM
Wow! Joshua / really good post / excellent explanation of what is about to happen while all this goes on behind the scenes / Thank You :_)
JayR Cela
April 22nd, 2008 at 3:28 PM
Is the fix for SVC-1896 going to actually stop that stupid shouty diagnostic?
April 22nd, 2008 at 3:31 PM
How about introducing performance based pay? SL works, the employees get paid? Always a subtle bug or a component not fully tested. Thats internal issues and more human based than hardware.
Yes, its a large and complex network and yes problems will occur. Sadly they always occur which infers the Testing model is flawed and/or the left hand doesnt know what the right hand is doing.
But its good start seeing Blogs on technical issues suddenly being open for comment now. Was there a memo sent around internally for such a policy change? Realisation that when competiton comes, lots of paying people will go? We all get sick of the “mushroom” syndrome.
The NEXT big step will be to see some Lindens actually reply (hint: follow Sidewinder’s example.)
Another good policy change would be to increase the stripend to try and keep paying customers from walking. It won’t hurt the SL economy, thats already in tatters from the constant daily crashing of much of SL. Useless things get fixed quickly, core customer items, after months, the problem still exists. So give the paying punters something, its not rocket science, indeed its simple marketing to keep the paying support base. Meaningless platitudes “like we thank you for patience” mean nothing when its a captive audience with no choice.
To end: Torely kindly make a “happy video” for general users on how to file a JIRA complaint and a Sequel “How to follow your JIRA complaint up” and of course to complete the Trilogy, “What to do when your JIRA issue is ignored”. It should be a best seller. Keep it under 2 hours for each and have an interval to.
April 22nd, 2008 at 3:33 PM
[...] Rolling Restart for 1.21 Server Deploy Wed/Thu/Fri [...]
April 22nd, 2008 at 3:41 PM
Oh cool, so when I get online saturday. hopefully I will be able to use all of my inventory again, cause I miss getting to rez stuff, like products I used to be able to sell at my store. People say H4 doesnt affect asset server issues, but it seems like the asset server went nutz when they installed H4…. soooo maybe there is a connection?
April 22nd, 2008 at 3:42 PM
Well, I have a region still offline and has been offline for about an hour or more. Related to this deploy or not? I don’t know coz I can’t view the version number for an offline region…
April 22nd, 2008 at 3:45 PM
havok 4 has nothing to do with the assest servers, it’s a physics system that runs only in the sim envrioment. it’s not connected to the asset servers in any way or form as it doesn’t fetch from the asset servers. again with its a physics system.. any of this registering?
April 22nd, 2008 at 3:46 PM
@11 Alf Lednev’s post.
I applaud that post so much, you speak so much common sense. And I fear the Lindens will not do what you say cause it makes too much sense, they always claim to take our feedback seriously, yet they always seem to do they’re own thing.
*sighs and falls out of chair losing hope for SL to show its true potential*
April 22nd, 2008 at 3:53 PM
@15 I appreciate what you are saying, and accept its truth… BUT the simultaneous deploy of server 1.20 AND client 1.19.4 has clearly confused the two in the minds of many people. Not everyone is a techy geek, in fact probably the majority are not. Therefore people will always associate Havok4 with the degradation in performance that we are seeing here.
April 22nd, 2008 at 4:00 PM
@13 Blaming the Havok 4 server deploy is akin to……
Blaming the fact that your automobile now is running poorly, because you forgot to fill the windshield washer reservoir with cleaning solvent.
Give it a rest
JayR Cela :_)
April 22nd, 2008 at 4:15 PM
Has anyone else noticed that since Katt started the blog posts have been all opened for comment and that generally that is calming the masses a little
Lots of good information in this post, slowly we piece together the technical details.
Keep up the great work towards stabilising things
April 22nd, 2008 at 4:17 PM
“Therefore people will always associate Havok4 with the degradation in performance that we are seeing here.”
Why do you think they call it Havok?
April 22nd, 2008 at 4:36 PM
Cat
April 22nd, 2008 at 4:53 PM
Thanks for the well detailed update and pretty fair transparency about issues you experienced while trying the new stuff.
Hopefully this will get us closer to some stability with the newer population numbers!
April 22nd, 2008 at 5:33 PM
Would be nice if somewhere at the end it said “we are working hard to improve Secondlife and thanks for your patience and understanding” seeing as these restarts are almost weekley these days and no noticable improvements in world.
April 22nd, 2008 at 6:02 PM
@10: Argent Stonecutter asks:
“Is the fix for SVC-1896 going to actually stop that stupid shouty diagnostic?”
Yes! I can’t swear we caught every instance, though. Send your thanks and requests for bears to the newly minted Infinity Linden. Still has that “new linden” smell.
@11 Alf Lednev comments:
“Always a subtle bug or a component not fully tested. Thats internal issues and more human based than hardware.”
Just trying to be transparent here - you’re going to hear about problems that may not actually be seen by most residents. I agree that our testing is not ideal. We’re working on it.
“Was there a memo sent around internally for such a policy change?”
Nope. Personally, I’ve been too swamped lately to post, having to beg and plead for others on my team to do it while I run around crazily. Every Linden has a different mix of technical and communication skills. We don’t route everything through a single voice of communication, and I know most technical folks here wish they had more time to post.
“The NEXT big step will be to see some Lindens actually reply (hint: follow Sidewinder’s example.)”
Heh - Sidewinder has a bit of a luxury to be nearly 100% focused on a single project. I agree, he’s doing an awesome job. Now where did I leave that cloning machine…
April 22nd, 2008 at 6:06 PM
Test it again. Code updates tend to be unstable when LL rolls them out.
Also, I thought rolling restarts were to be confined to one day a week not three days of the week. While the TOS allows LL to have unscheduled interruptions of service, just because you can doesn’t mean you should.
With all the disruptions recently, minimal outages for updates should be the rule of thumb right now.
April 22nd, 2008 at 6:22 PM
I don’t know about the rest of second life, but in Withernook I can no longer wear clothes or attachments since the “upgrade.” I have to go to an adjacent region with the “old” simulator to put on clothes — including my hair. This happens on land I own, on land that my group owns, and on public (linden road) land. THIS SUX BIG TIME!!!
April 22nd, 2008 at 6:42 PM
OK now we are getting somewhere with the info,,Thank You LL.
If it was me I might have taken more time between steps to see where the bugs would pop up and anybody who has deployed alot of severs knows there is always something. Good Luck and thanks for trying to keep us in the loop this time.
April 22nd, 2008 at 6:43 PM
@Joshua thank you for giving us a detailed report.. the jira links to specific issues are also very useful for folks who may not read the jira often
Is this more or less the same build that was on the beta grid before it became unwell?
April 22nd, 2008 at 6:54 PM
Objects not rezzing, scripts not loading, come on get on it please.
April 22nd, 2008 at 6:59 PM
I appreciate the report as much as anyone else, however, I am very displeased with the way step two of this process is being carried out. My region just restarted with zero warning. Some of us are actually trying to conduct business here, and I don’t think it would kill someone at LL to issue a warning to the regions about to be affected that they are going down in 5 minutes. Now I notice that those affected in step One got a Blog entry saying 184 regions would soon be affected… but I don’t know if those Sims received any notice before their restart either. This is really unacceptable. It stinks of LL just assuming that Restarts have become a common enough occurance that no forewarning is needed anymore… and that is a sad state of affairs. I certainly hope that this was a simple oversight, and not the new status quo ~ but I know for a fact this is the second time in under two weeks that my region has been restarted with Zero Notice, both times while my business was running events. Needless to say, I don’t apprecaite having all of my patrons & staff booted from SL, nor the sudden termination of our events without even a chance to redirect ahead of time.
April 22nd, 2008 at 7:16 PM
I’m curious which sims have been updated, so I can try them out. I saw some behaviour on the Beta Grid, which might even be worse than SVC-2141. Whether it’s better or worse would be a good thing, as narrowing down the cause. I’d like to try it under real-lag conditions.
Maybe I should Jira for a Lag Simulator, in Beta Grid?
April 22nd, 2008 at 7:54 PM
@23 “Would be nice if somewhere at the end it said “we are working hard to improve Secondlife and thanks for your patience and understanding” seeing as these restarts are almost weekley these days and no noticable improvements in world. ”
You’re not looking hard enough if you’re not seeing the improvements these folks are making.
April 22nd, 2008 at 8:21 PM
Look harder Says:
April 22nd, 2008 at 7:54 PM
@23 “Would be nice if somewhere at the end it said “we are working hard to improve Secondlife and thanks for your patience and understanding” seeing as these restarts are almost weekley these days and no noticable improvements in world. ”
You’re not looking hard enough if you’re not seeing the improvements these folks are making.
Oh I look every hard it is you that is missing the point!!!
Is not always what you do but what you say!
April 22nd, 2008 at 9:23 PM
Thank you for your prompt response to my phone call and your roll back — all attachments happy in the old version now.
April 22nd, 2008 at 9:43 PM
You don’t beta test these first? lol
That’s twice this one has messed up. Are your coders on three strikes and out? Should be…
April 22nd, 2008 at 10:50 PM
Well, I do not understand those who thank SL for detailed info, amazing future changes, etc. Technical details are always welcomed although not as much as clear statement: “SL is off”. There is lot of us who have absolutelly no time to “decode” your messages. Next time go directly to the point, please. By the way three days off means significant loses for many business people around in here. It would be only fair at least not charge them proportional amount out of monthly fee. But it will never happen - or am I wrong?
April 22nd, 2008 at 11:12 PM
For God’s sake if you can’t finish a rollout by the middle of the week or thursday, don’t start them. If you get delayed, don’t soldier on into the weekend and then split for Sunday leaving an unstable disaster behind like the last time.
April 22nd, 2008 at 11:24 PM
Thank you for the well laid-out and explanation of what you’ll be doing.
Communication like this really does show a bit of just how complicated things like this can be. Plus, I like being treated like I”m smart enough to grasp what’s going on instead of just being told “we’re banging n things”.
Keep up the detailed explanations, checking ‘em off as completed, and good luck with the roll-out.
April 22nd, 2008 at 11:28 PM
Sometimes I feel like a Guinea Pig….:)
I am sorry to say that as of late i tend to log in less when a deploy is under way. As for losing touch with my friends in SL, well, that is independent of the technical problems. Some people, like in RL today, simply get bored and find new “toys” to play with. How sad!
April 22nd, 2008 at 11:35 PM
Thanks for the info Joshua.
Just curious…
Does the new M have a license to thrill?
April 23rd, 2008 at 12:20 AM
Once more, the roll back of the previous roll back !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Be serious Lindens………
Or code a new full featured Pacman
April 23rd, 2008 at 12:33 AM
Hi Josh,
the positive thing is, that you tried on one rack first before rolling out to the whole grid, if I understood this correctly. As I see from fellow players, grid is or was offline anyway…
But one thing is still not clear to me: Don’ t you have a so- called “sandbox- solution” in the Labs, where you can test in a sort of “isolated test- grid”, that new versions have correctly working facilities? Or are new server versions only simulated mathematically without such kind of testings, before rolled out? Don’t know whether this is a silly question.
Good luck - Lara
April 23rd, 2008 at 1:01 AM
@43 Laraya:
I will take the liberty of answering for Josh:
Yes, they do have a sandbox, that is the beta-grid, but since there are not many ppl using the beta-grid, some issues don’t show untill the code goes into the main-grid, with 60,000 users online.
Testing software is generally a difficult task, SL is much more complicated than most software, so it’s an impossible job to test for everything before releasing new versions.
Don’t worry… be Happi
April 23rd, 2008 at 1:24 AM
@Joshua………….LMAO i just read your post……….and you hit the nail on hte head……..
“Heh - Sidewinder has a bit of a luxury to be nearly 100% focused on a single project. I agree, he’s doing an awesome job. Now where did I leave that cloning machine…”
Hmm shouldn’t it be so that people focus on one project and get that right first……….WORTH THINKING ABOUT I THINK
Basically in my opinion this is the root cause why sl is borked at the moment…………nobody is concentrating on one single thing, instead trying to deal with numerous problems at the same time………..you guys need some serious help with planning if you ask me.
LOL…………and here we have a Linden openly saying they are doing to many things at once
Even though the proof of the pudding is in the eating……see H4 and sidewinder……..except for a few small things it works perfectly 
Lindens get real you can’t even see a solution when its staring you in the face
April 23rd, 2008 at 1:28 AM
oops i forgot all the TM….bla bla trademark bits
April 23rd, 2008 at 1:52 AM
Yes, let’s keep the Havok 4 borks as they are: llTargetOmega, half your torso disappearing in walls and ceilings on collision, sliding doors not opening, the step height that means you can’t walk up lots of shop steps now, and go flying/falling at the top of stairs/stumbling over very low objects, sim crossings, weird vehicle behaviour etc etc. Bork free Havok ain’t, in fact causes me way more problems than the old Havok, and sim crashes affect me hardly ever, I doubt I’ve had one a month in all my time.
And Windlimp, mmm, well apart from rushing out to buy a new computer (NOT) to see a blue sky instead of a grey sky, and fps falling from 20 to 0.4, and all the extra data added on to already loaded asset servers, plus the extra capabilities of Havok, taking up more data on, er, already borked and overloaded asset servers, combined with new server code, which also lets Mono on, more scripting capabilities which, er, pushes even more on the asset servers, oh, and change the client GUI too, do it all at once so you have no clue where to start looking for bugs, and you kinda have exactly the same mess we are in now. It cannot be coincidence that the grid started falling over big time as soon as these changes were put in for the un or insufficiently tested Havok and Windlimp.
I mean, Sidewinder Linden himself admitted in his blog post, that he made the call to still release it all with big problems, so I wish the Windlimp and Havok fan boys would STFU on this. The Linden responsible has admitted it had flaws and made the call to release it anyway!!!!
If M (Talk About Pop Music) Linden (there is a French Pop Singer called M, no ™ for you lol) is reading these, then for God’s sake read the year old Open Letter and fix what we have been asking for repeatedly to be fixed, and STOP all these ruddy unwanted shinies that just bork up the system even more. At least until you have a strong, stable, pretty much bug free system, with enough spare capacity to handle it. Then you might not see your active users and premiums evaporate away before your very eyes. Oh, and next time Phil wants to say we are all a bunch of poor rural stay at home pathetic nobodies, then maybe you can gag him, or give him a ruddy big pink slip from us all… Get with it Phil, more people buy laptops than desktops, mobility is the future, haven’t you heard that yet? Don’t have a mobile phone Phil? And they rather like WiFi. Except, we know it isn’t our kit or configuration, but borked and badly written code. That’s why every other update works, and every other one breaks it all again.
@ 44 Yeah, well when Sidewinder went on about hundreds of hours of testing, and I said how could you when there were major borks like rotation, how were they not picked up, he admitted on this very blog they wrote it off as a small thing and pushed ahead anyway!!! It’s not difficult if you do it properly, have a proper system set up, know which code base you are adding your shinies too, and have a far better way of reporting things that the laughable JIRA. At the moment all they generally get is fan boys reporting, who are loath to criticise because they are the 1% geekoids, and so it’s not properly reported on. And the fault of that is on management, or rather the lack of any, and that is all at Phil’s feet.
I mean, if that’s what happened at Real, look what happened to RealPlayer lmao.
April 23rd, 2008 at 2:40 AM
@ all the flamers and whiners … Oh yes … flaming lindens like a pack of 2 yr olds in dirty diapers is really productive! Frankly I just want to say to keep up the work trying to push this code out … take as many half steps as you need to … making it better is likely going to be step-wise, not done in earth shaking leaps of imrpovement. Keep us informed please.
April 23rd, 2008 at 4:14 AM
How about the fix for the 30 second crash due to something in the upgrade that the firewalls dont like? Not only cant I stay on line but when I went to install the next release for Dazzle etc it wont accept it and reverts back to a previous install.
Oh, and while your at it, reports of Meeps and elevator/ transportor failure after the shift to havoc 4 upgrade on the islands. oddly, they work on the test regions of havoc 4, but not the old islands that were upgraded, and even when deploying new ones to replace the ones that did work but dont, they still dont.
steps can only go so high!
Really would be happy with back at version 11. that went well.
Less foam, more steam!!
thank you
Kath
April 23rd, 2008 at 4:51 AM
Hi,
Personally, I am appreciating this post.
Not trying to be antagonistic ok? But I think you see a lot of “it doesn’t work” posts here is that it doesn’t seem that so many issues are addressed when using the other avenues available.
Now don’t get me wrong. I really DO appreciate the live chat system and I have had many resolutions to issues that way. Just today I talked to a live chat tech about TP failing and they were blaming my connection and / or the sim or sims I was on. Well, I had all greens in the indicators. I run over a spiffy fibre optic connection that is normally solid as a rock.
It’s seems routing between certain sims were the issue as I could “bunny hop” through certain routs but not others. Interestingly enough, there seems there MAY be a link between this issue and the different versions of the simulator code they are each running. Maybe a connection there?? Interesting that from reading the blog here, I’m not the only one seeing the degradation in the TP functionality. Bottom line though, weather it’s the physics, the asset servers or the hamsters need feeding… I sure would like to be able to get around again
Also, I’m in great hope that when hovering in the air, such as on Egill, the slow sinking back to the ground or nearest non phantom item will stop.
Last but not least, since the last server code change, doors and prim detectors in Egill have begun to fail or act strangely. Some prim detectors have just stopped working. Doors have ceased responding unless I am online.
Hoping for the best in the near future!
Enysy
April 23rd, 2008 at 5:06 AM
@45
first there has to be a common objective and a goal, a clear strategy determined by the management, then teams are created who will focus individually on dedicated areas and projects based on the team skills coupled to a clear deadline! Regular teamleader meetings must be held to discuss progress or bottlenecks ->the senior management is informed weekly, so they can steer or take action where needed.
->instead of posting then highlight and lowlights (warm & cold) on a daily basis, the management team should report on a weekly basis through KATT LINDEN the status of the objectives towards us users!It has no sense if one day sidewinder reports here, or joshua or anyone else … it has no value cause we dont see the big picture!
teams must be challenged on a regular basis, to determine/evaluate there performance/progress in time! Steering of those groups is essential cause if one of those teams is gonna loose track or one of these teams are going to follow there own path —>it will go wrong !
now after analyzing all the problems since weeks, I can conclude that a harder management approach will be necessary at lindenlabs! I still have the impression that lindenlabs employees act like volunteering charity workers than as professionals! but hey, lindenlabs is a company who wants to make profit like other businesses do!
I know I am critizing a lot lately, but I run a company myself in rl and gave my 2 daughters (they are nicer in rl than in sl, dont think my daughters have cat ears or a tail in rl
an island where I invested 7500 US$ in one year for there pleasure and fun! for some it is a lot of money, but as long if my kids enjoy it, I am happy with that! benefit is that thx to sl i can communicate with them from distance cause I work 97 % of my time abroad!
but hearing my kids moaning and complaining more and more about the poor performance of sl, is making me as a dad and a professional going to the roof !
Again I hope that the new CEO will take actions short and longterm!
cosa
April 23rd, 2008 at 5:19 AM
@45
“see H4 and sidewinder……..except for a few small things it works perfectly”
“Few small things ?”
like :
- Hugs don’t work anymore,
- Cars don’t work anymore, (money wasted, unless you just need cars to decorate your garden)…
- Voice doesn’t work anymore… (but is has been decided that it was a “useless” feature, so… who cares ?)
AND I am not a scripter nor a builder… I guess there would be much more to add to my list…
“a few small things”….
HA !
April 23rd, 2008 at 5:32 AM
Joshua,
This didn’t go well, that’s obvious, and I don’t think anyone is happy about that. There is a LOT to complain about with the way LL has handled things over the last year and especially the last month or two.
But let’s also be honest about this: kudos and credit where credit is due!
One thing that a great many have complained about (and rightly so) is communication and transparency, or the lack thereof, on LL’s side of things.
Your post is a BIG step in the right direction, as is your replying!
No, the deploy didn’t go well. Yes, you had to post hot/cold info on the same day.
BUT YOU DID COMMUNICATE, CLEARLY AND FULLY.
Bravo! Thank you! I am starting to have some tingles of hope about LL again in the last couple of days, and this blog posting of yours is one of those “tingles”.
I just hope now the next time you make a nice detailed blog post, it’ll be with good news.
[thumbs up]
April 23rd, 2008 at 5:33 AM
@50 myself …..
update on last alinea
but hearing my kids moaning and complaining more and more about the poor performance of sl (knowing that I pay 295 US$ for a service that should work) is making me as a dad and a professional going to the roof !
cosa
April 23rd, 2008 at 5:45 AM
@15 Masuyo is incorrect that Havok 4 has no connection with the asset servers. Here is how it is connected:
If Havok 4 is more efficient in running physics, that allows more time to do everything else. That everything else includes fetching stuff from the asset servers. Therefore the load on the asset servers increases.
In an interconnected system like Second Life, *everything* affects *everything else*. That includes you players. I suspect when originally designed, SL assumed avatars might carry some tens of prims in attachments - boots and a sword for men, for example. Talented designers have created complex hair, jewelry, shoes, etc. so its not unusual to see someone with 1000 prims on their body, all of which has to be fetched from the asset database.
So, if there is anything useful to suggest here, its when one item is changed, think through how that change will affect the other parts of this connected world.
Prime example: Mono - it will drastically speed up how fast scripts run. OK, so what *else* will a sim be able to do with the extra time, and what loads will that put on the other parts of SL??
April 23rd, 2008 at 6:26 AM
Joshua - thanks for the info. Curiously, as a techie, it gives me MORE confidence that you guys are making progress and do know what you are doing lol. Kudos. And to echo others - good job opening up the posts lately.
April 23rd, 2008 at 6:44 AM
@51….hmm Vic by a hug AO not a free one my friend……will work believe me……….not gonna list all the ones that work here
Hmm cars…….no idea about this one but i presume the same will apply……buy a car from a decent builder and it will work ……builders that care will have informed themselves over the changes caused by H4 and issued updates………those that didn’t well………..you have to ask youselves why not
Ok to the point…..all i was saying was joashua has openly admitted everything we residents have said for months…..they all are working on to many things at once………….that that doesn’t bring much in the way of results is bloody obvious
and yes i build and can script Vic…..H4 does work…..there are known issues…………..but not as big as….transactions failing……tp up the swanny etc etc the list goes on………but once again…………THESE HAVE NOTHING TO DO WITH THE PHYSICS ENGINE H4
But believe me Vic none of these have anything to do with H4……
April 23rd, 2008 at 7:00 AM
34 SecondLife - How To Make Money In Second Life » Rolling Restart for 1.21 Server Deploy Wed/Thu/Fri [DELAYED] Says:
April 22nd, 2008 at 9:14 PM
[...] post by Joshua Linden [...]
Please excuse me but isnt this a conflict of accuracy?
I know far too many vendors, builders and designers who are and have been loosing money hand over fist with these continual starts, errors, stoppages and what ever else you wish to call them.
I may be over simplifying things be asking, “Wouldnt it be easier and possibly better to sort out the problems on existing versions instead of continually releasing these untested beta vesions? Let those who write the programmes be the testers and pay them on performance has as been suggested above then see how disgruntled your CUSTOMERS are Lindens.
I was bought up to believe; “The customer was always right. The customer was the life blood of any organisation.” Probably worth keeping in mind that Happy satisfied customers (members) dont wander. (they dont have a reason to do they?)
I just know I am only one who went to 3 different sims in one day and each was rebooted and then it took me almost 3 hours before I was able to get back in to SL after being kicked out.
This is becoming most frequent as well as most inconvenient to many.
AS I say, maybe I am over simplifying things.
April 23rd, 2008 at 7:20 AM
I spent more than 50 000Linden$ on cars, who worked perfectly until H4. Having spent that much money for items that don’t work anymore, for me, is a transaction failure.
I just don’t see the point in spoiling things that did work perfectly before H4.
Who should I contact for an upgrade of Voice ? It is ruined. Who could fix it if not LL ? LOL
“builders that care will have informed themselves over the changes caused by H4 and issued updates………those that didn’t well………..you have to ask youselves why not ”
LL “built” Voice, and apparently don’t care about maintaining a good service of this feature.
I am asking myself why not…
You’re right…
H4 is the biggest mistake ever.
It is my point of view, anyways.
April 23rd, 2008 at 8:57 AM
I am just tired of part of my outfits not rezzing. I have 3 beautiful gowns I can’t wear, because the skirts won’t rez or attach. (Sob)
April 23rd, 2008 at 9:35 AM
Josh thanks for the reply, You’re so good about that.
/me takes out HIS cloning machine and makes more Josh Lindens!
April 23rd, 2008 at 9:38 AM
Yes, H4 killed some vehicles. That is not in dispute. But you might be interested in knowing why.
H1 had bugs. Scripters created workarounds because of those bugs. Vehicles then could work nicely.
H4 fixed H1’s bugs. Workarounds no longer are needed. The bugs don’t exist, after all. Vehicles with that sort of scripting broke.
H4 FIXED bugs. That’s a good thing! The scripters who keep current with changes have created updates, fixes. THOSE are the people to be getting your products from. Don’t think that your freebies or those made by scripters no longer with us will be automatically fixed because you acquired them once upon a time.
Just as a reminder, people purchase vehicles in meat-space…real-world, etc. And guess what? They don’t work forever. And they have to purchase new ones at some point. Did you really think that your SL car would last you forever?
April 23rd, 2008 at 9:43 AM
Thanks for the update. The improved communication and increased transparency is nice to see.
I’m much happier with the way these updates are handled now compared to when I started SL. Seems that too many people have either forgotten or weren’t around when the grid used to be taken completely offline for updates and problems didn’t become apparent until it was switched back on. I shudder just thinking about it. LOL
Keep up the good work. Cheers!
April 23rd, 2008 at 10:26 AM
@57
“Wouldnt it be easier and possibly better to sort out the problems on existing versions instead of continually releasing these untested beta vesions?”
Valid point. Nine out of ten “aquired” people leave SL very soon and do NOT stick to it for more than a few hours. I would be interested in a survey there, what the reasons for 90 percent of visitors leaving it before starting it exactly are. But i am pretty sure that the fatal errors in the client software and the steady instability of the grid do have a lions share.
Rosedale stated that SL users only have in common that “they have a lot of time”. Indeed, you must be extremely patient while dealing with all the preferences tweaking, crashing, rebooting, total system lock ups…the only way to avoid it is standing still somewhere enjoying the picture of the gorgeous windlight sunset on 5 fps. Well, i an have some nicer sunsets in RL without all the fuss.
April 23rd, 2008 at 10:36 AM
Joshua….you are getting the idea…we love to hear whats happening and we appreciate it. PLEASE try to devote a few minutes to posting in the blogs….this really can make a difference. I am sure everyone knows that you guys and gals have a huge mess to fix and that it is tough getting all this negative stuff on the blogs….but understand it from this side of the fence…..all we see is problems….no one tells us that they even know what they are or how they hope to fix them. This leaves ppl with the thought that it may not be repairable …. and now anger has gone to apathy…… the next step will be EXODUS. This has to be stopped ……. you know it, we know it !
April 23rd, 2008 at 10:51 AM
I hate to think what all the ill-informed complainers would do if the Lindens suddenly decided to pull the plug and close their business. RL can be a very cold and unforgiving place….
April 23rd, 2008 at 10:54 AM
Is the e-mail bug back again?
All of our vendors are not functioning right now, only the ones in the same region as the server…. jesus…
April 23rd, 2008 at 10:55 AM
@everyone
by the way, it is remarkable between the announcement from the new CEO yesterday and now, there havent been any new BLOG SUBJECTS added from LINDENLABS!
this probably indicates that someone has read the reactions from most users and understood clearly that spreading HOT & COLD messages doesnt help LINDEN moving forward! Which ofcourse I can only encourage!
I hope they use there energy with finding a clear solution finally for all the problems caused due to lost focus on the whole picture ! Regrouping on a battlefield can in many cases give result !
Maybe the new CEO will be the start of a new beginning for the grid!
mmhhhh I am getting positive
cosa
April 23rd, 2008 at 11:00 AM
@66 Well said and well done. Wish all could ‘get positive’ and enjoy something (SL) that they might not otherwise have :o)
April 23rd, 2008 at 11:09 AM
I enjoy SL® everyday, from the Real World or “Meat Life” as was coined by someone here. I get extreme pleasure reading about all the troubles in-world. I don’t get in-world much nymore as there is really nothing constructive you can do in SL anymore, it’s the biggest waste of time since the Republican Party was invented. Well that’s a subject for another post.
On Topic:
Server code can and should be implemented at any time of the week if it has an adequate staff trtacking the results.
April 23rd, 2008 at 11:44 AM
I’m glad to see this is back on schedule. The sooner all these changes are rolled out, the better. (assuming Intermezzio works out :S )
April 23rd, 2008 at 11:48 AM
@18…
Quote
“13 Blaming the Havok 4 server deploy is akin to……
Blaming the fact that your automobile now is running poorly, because you forgot to fill the windshield washer reservoir with cleaning solvent.
Give it a rest
JayR Cela :_)”
Totally misunderstood I’m afraid. It’s not a question of filling the reservoir with solvent - It is a question of the manufacturer wanting to modify your new car with stuff you don’t really need, making it perform worse (i.e. engine trouble), and on top of that ignoring to fix the earlier engine troubles by carrying out further new looney modifications.
April 23rd, 2008 at 11:57 AM
Hurray! The issues I identified yesterday were addressed by LL and now my region starts in 1.21 and I can wear my hair and “other attachments.” Kudos for doing something right!
April 23rd, 2008 at 12:02 PM
@61 I think it would be much better for business to go ahead and take SL offline to do things, unless there is some emergency. Rather than have everything always in a state of unpredictable flux, depending on region, etc.
Never knowing what you are going to walk into is a much worse psychological paradigm than being able to know it’s working, or know it’s not.
And then, too, you always knew if you got on right after an update, whatever was going wrong was likely due to that update.
Now you don’t know if they did an update, a region-limited thing, or if it’s you, or what.
And before, you knew if you didn’t want to bother with these changes, just don’t log on till some hours or even the next day after an update. Now it’s hard to tell what is causing your problems.
***
Many thanks to whoever it was who decided to allow all blog commentary to remain open.
coco
April 23rd, 2008 at 12:05 PM
Failed logins and unable to tp to Journey Rock Band Island.. hello, anybody out there?
April 23rd, 2008 at 12:06 PM
it’s odd when LL do something right (ie testing server code correctly and refusing to deploy when bugs still pop up) the blog doesn’t fill up as fast, becuase all of the whingers can’t say anything bad,.
i still like SL, can’t see me stopping liking it anytime soon.
April 23rd, 2008 at 12:08 PM
I’d have to agree with Cocoanut - take it down! Spinning our wheels trying to figure out if our viewer is screwed or sl is wacked, esp with no live status page to allow us to know what’s happening and when. Maybe KAT or the new CEO can start to talk to us, sooner than later..
April 23rd, 2008 at 12:11 PM
@70 To follow your argument would then beg the question, “Why would you want to keep and continue to drive such a car?” Not sure what your point is :o)
April 23rd, 2008 at 12:21 PM
The Linden’s got a new boss so like any good employee’s they are going to let us know a lot more and tell the new CEO, oh yes, we been doing this all along. It’s the old suck up to the new boss and cover your ass routine and don’t tell the boss everything that’s going on. In any company, you always tell your boss everything is running great. It Keep’s him off your back while hopefully you fix it before he sees you really don’t have a clue what your doing. But, he will figure it out soon enough and him hopefully get this SL on the road to recovery. Hopefully this new CEO is a head chopper and will also get rid of some of the dead weight.
April 23rd, 2008 at 12:26 PM
Probably already said. But it seems that last night broke the object to object email. And when sat on a rotating prim you won’t rotate anymore… :S
April 23rd, 2008 at 12:28 PM
So now that the script that was working on my house to let in only my household is broken. Apparently now anyone can open and close the door
Does this mean I am screwed if the creator is no longer in SL? If this is going to keep happening with LL updates, ppl are going to stop buying as many scripted items bcz they are afraid they will be worthless in 90 days.
April 23rd, 2008 at 12:36 PM
@79 Let’s see, we went from Havok1 to Havok4. And this is something that happens every 90 days? Ha! Why do I bother….
April 23rd, 2008 at 12:51 PM
Remember to send sim warning before restart, please!
April 23rd, 2008 at 1:05 PM
Great post Josh! Really helpful. And welcome to the new CEO.
Here you will always receive a large glass of ‘whine’ for your efforts.
@77 me thinks the ‘new guy’ probably has access to the internets and can see for himself the comments from resis - good and bad.
I’m not happy I lost my much-loved customized AO (with 32 no-copy poses in it!!) 2 weeks ago due to a ‘data glitch’ but this will happen in any environment that is continually under development.
Calling for a stop to new features and code is like trying to stop the future.
The possibilities of virtual worlds are enormous but if you wish to live and trade at the sharp end of technology, there will be times when it all goes pear-shaped sometimes.
We’ve come a long way but the journey isn’t over.
Think of the stories we can tell our grandkids via holographic speed-of-light worlds in the future about the ‘early years’ of this emerging technology.
April 23rd, 2008 at 1:24 PM
I’m having attachment issues, failed tps, extremely slow rezzing on a sim with 3 people on it and so on. This is ridiculous.
April 23rd, 2008 at 1:28 PM
“I hate to think what all the ill-informed complainers would do if the Lindens suddenly decided to pull the plug and close their business.”
Why should they shut down as long as the complainers pay for their “cutting edge” service?
April 23rd, 2008 at 1:37 PM
@84 Indeed ;o)
April 23rd, 2008 at 1:43 PM
@84
You don’t shut down, you sell to the highest sucker, er, bidder.
April 23rd, 2008 at 1:50 PM
@85
Would YOU buy THIS? I won´t. I would hate meeting myself every other day on the blog.
April 23rd, 2008 at 2:03 PM
@87 Would YOU buy THIS? I won´t. I would hate meeting myself every other day on the blog.
?
April 23rd, 2008 at 2:05 PM
well, my alt then, he he. complaining.
April 23rd, 2008 at 2:15 PM
I think I see what you’re saying. I’m actually quite impressed with the patience of the Lindens and how thoughtfully and diplomatically they respond to complaining. Tells me how much they love their ‘baby’ that they would want to continue to help it grow despite the slings and arrows :o)