Simulator Stability Improvement And Crash Rate Reduction Results
Thursday, April 10th, 2008 at 5:02 PM by: Sidewinder LindenSix months ago we announced the beta test process of a new Second Life simulator, with the explicit goal of improving simulator stability. The new simulator went through six months of public beta testing, was in service on over 635 Second Life regions throughout the beta process, and during the public beta Early Adopter Program was found to dramatically reduce simulator crashes.
In addition to improving the Second Life experience by reducing the number of times a region is restarted (which forces all avatars off the region at the same time), lowered simulator crash rates also reduce content loss by reducing the number of times a region must be “rolled back”. Whenever a region is rolled back, any content changes since the last snapshot are lost, and the increased stability of the new simulator has likewise reduced the number of region rollback, and thus reduced the rollback content loss rate substantially.
Now that we have a week of active experience with the new simulator throughout Second Life, I thought it would be good to share the crash rate details so that you can see the impact of the Havok™4 project team’s work on increased simulator stability.
Below are two charts to show the stability improvement provided by the new simulator. The first chart shows the number of regions that crashed across all of Second Life, per day, due to physics engine crashes. Physics engine crashes were the most common cause of region crashes, and have been reduced by 97.2%
The second chart shows the number of regions that crashed from all crash causes. We also resolved some non-physics crashes during this project, resulting in a 68% reduction in region crash rates, across all of Second Life due to any cause.
We are continuing to resolve open issues with the new simulator, to make it even better both in terms of crash rate and functionality, and will be posting updates as this work is done.
Thanks again for all of your support and involvement,
Sidewinder Linden
Havok™4 Program Manager
April 10th, 2008 at 5:12 PM
CONGRATULATIONS !!!!!!
Well done Havokk 4 team, and thanks for the great communication Sidewinder.
Thanks agains !
Woooooooot
–Anthea
April 10th, 2008 at 5:13 PM
well thats what H4 was puspose to do
Great work!!!!!!!!!!!!!!!!!
Usagi
April 10th, 2008 at 5:14 PM
The crash rate is lower because for over one month now the Crash Logger is not working and we are being Logged out so that doesn’t count as a crash.
I don’t think data is valid.
April 10th, 2008 at 5:14 PM
Congratulations, Linden Lab (and Sidewinder and Andrew).
April 10th, 2008 at 5:15 PM
What about Class 4 vs Class 5 sim crashes?
For the people who have sims running on older hardware running the new physics engine, are there slow downs or more crashes versus having it run on Class 5 hardware?
April 10th, 2008 at 5:16 PM
@3 Totally Fooled - They are talking about sim crashes, not client viewer crashes.
April 10th, 2008 at 5:16 PM
THIS WAS A TRIUMPH
I’M MAKING A NOTE HERE
HUGE SUCCESS
p.s. >>3 you’re thinking of viewer crashes, not server crashes. the viewer crashes more than Windows ME currently.
April 10th, 2008 at 5:20 PM
Very dramatic results! What’s next?
April 10th, 2008 at 5:20 PM
@3 Totally: The crash logger component that was not working was the viewer (client) side. The server side crash logging is fine, and this data is for the Havok4 project which is a server-side enhancement. /Sidewinder
April 10th, 2008 at 5:21 PM
@3: if that was about *Client* crashes, I’d agree, but this is a bout *Server* crashes which don’t get reported (or not) by your client…the servers keep logs of their up- and down-time (simply put).
If we do criticize LL, and there is plenty of stuff to criticize them for, let’s keep the facts straight and refrain from nonsensical statements that only lower the critic’s (and this the criticism’s) credibility.
April 10th, 2008 at 5:21 PM
@5 Gattz: Class 4 vs Class 5 is an interesting question. I do not know the answer off the cuff but will look at the data. My suspicion is that this is independent of server class, but I will check to be sure. /Sidewinder
April 10th, 2008 at 5:21 PM
well it was longer for me! the crash logger wasnt working for 6 weeks.
April 10th, 2008 at 5:22 PM
Yes Well Done Havoc4 Team . I am very impressed with this server release. On a regular daily cycle our Sim would crash at least once a day . With havoc4 we were able to run 5 days without a restart and without a crash. Aside from the minor glitches that will soon be fixed i am very happy with the results. Thanks Sidewinder for your dedication and hard work on this project!
April 10th, 2008 at 5:23 PM
@12 U M: The crashes reduced by this project are server-side crashes. The physics engine for Second Life runs on the servers at our data centers and is part of the server simulator code. That is the area that this project was focused on for crash removal. /Sidewinder
April 10th, 2008 at 5:29 PM
Impressive graphs! And not only because they show the improvement, I’m als a bit amazed by the fact that previously 1500 sim crashes a day was quite a normal figure! Then again, I’d need to see it in perspective, 1500 is a lot, but how much percent of all running sims is that.
Anyway, congrats on a job well done
April 10th, 2008 at 5:30 PM
The sim crash results are fine. Great!! Sidwinder, you are the best.
Now please take over viewer development.
April 10th, 2008 at 5:31 PM
in the meantime clients are crashing people left and right if they are using it for more then a few hours at a time without relogging. Memory draining is the key problem.
ALSO rotation scripts seem to stop working all together unless you re-rez that object.
Other then that yes sim performance is 100% better…
April 10th, 2008 at 5:32 PM
@15 Daedalus: I don’t have the exact number at hand, but the total region count these days is somewhere around 17,000 last I heard… If you’d like an exact number I can find it, but this is probably close enough for the rough percent you were looking for (I hope). /Sidewinder
April 10th, 2008 at 5:34 PM
@17 william; The viewer teams are working to reduce viewer crash rates.
We do know about the llTargetOmega issue (causing random stops of rotating scripted objects) and have invested days of development time so far trying to find good ways to create and reproduce the problem, as well as to find proposed fix locations. This one is elusive because it is so infrequent, happing every hour or day, which makes it very hard to hunt down. We will find it, and are actively pursuing the cause.
/Sidewinder
April 10th, 2008 at 5:35 PM
on 04/05/08 SL has crashes?
Your kidding, right? For your remembership, 04/05 was saturday with the “big offline”.
I believe, that the sim stability with the new server is better, but i believe also, that many other problems comes together with the new server version.
Maybe you’ve read the blog the past week, we have had alot of problems with logins, TPs, transactions etc. All this failures are based on the database / asset server and not the simulators. But if many things borked and not usable, this will decrease the crashes also. So I’m not sure if this statistic is a true one. Let’s talk again, after the main problems with the database is fixed.
I hope things will be better soon, because i have round about 60% lost in my business with all the problems (transactions, failed TPs, downtimes - and yes, i know it was the provider, not LL)
April 10th, 2008 at 5:35 PM
@16
Please take over the whole company …….they sure need a lesson in customer service and Sidewinder and Team are the only true professionals there. Well done Sidewinder and the Team. Pity I wont be staying in SL long enough to appreciate it, after Jacks attrocious handling of SIM pricing.
April 10th, 2008 at 5:36 PM
There are still some quirks that need to be ironed out with H4, but by and large… well done, Sidewinder and team!
Your group’s focus, progress, and above all communication are models that the rest of the LL programmers/groups can only dream about. Kudos, well done, and all that!
April 10th, 2008 at 5:38 PM
/me nominates Sidewinder for project manager on fixing the asset servers!!!
April 10th, 2008 at 5:39 PM
Man, then I would love to know what I’m doing wrong, because I crash more than ever now.
April 10th, 2008 at 5:44 PM
[...] out the results of the Havok4 crash reduction here. And see my previous post here. Posted in Second Life. Tags: havok4, Second Life, sidewinder [...]
April 10th, 2008 at 5:45 PM
@24 Ultra: Viewer crashes happen to one person at a time.. and I suspect that is the type you are talking about. The crash reduction provided by this new simulator version is on the server side. These crashes throw everyone who’s on a region off at the same time, and in many cases require a restore of the region configuration before the server will start again - causing lost content. There are other teams working on viewer crash rate reduction. /Sidewinder
April 10th, 2008 at 5:47 PM
SL data has no credibility because SL controls the data. Pretty simple.
April 10th, 2008 at 5:47 PM
I rarely crash, but suddenly tonight I have been crashing a lot which is scary. Just ironic that I came over here and found this blog post titled “Simulator Stability Improvement And Crash Rate Reduction Results”.
Thanks Sidewinder for the confirmation on the llTargetOmega. I think I’ll go make some bubbles and hopefully things will be better in a few.
Maddy
April 10th, 2008 at 5:47 PM
Nice work and excellent communication! Congrats. And Thanks.
April 10th, 2008 at 5:49 PM
@27 Wake up: I offer openly to discuss how this data is collected with you. I offer as additional evidence a discussion with region owners, and ask them how often their regions crash now, as opposed to before the update. Most in the Early Adopter program reported to me that they never or hardly ever see region crashes any more, which is consistent with the data. Owners of regions converted in the full rollout have been sending me messages with the same commentary, so I suspect that this data is correct. /Sidewinder
April 10th, 2008 at 5:53 PM
@26 ….. Sidewinder….with all due respect…the major effort seems to be on pushing through new features as per the latest Release Candidate with new Dazzle and other silly stuff. No one seems capable of fixing the basic issues …. and if they are working on it….why cant this be communicated to the thousands of frustrated customers out here? You are about the only person who responds to people…the rest of the blog entries are very rarely open to comment and never have responses from the teams involved. I am sure we would all like to hear what is being done re. asset servers and database issues in some detail, together with what is beingg done about all the other problems that have been around for so long. I could put up with being screwed over SIM pricing if I thought there was a chance of this system working properly in the near future. The impression right now is that all the activity is to get more people hooked and to make it all look financially interesting for an IPO.
April 10th, 2008 at 5:54 PM
That’s nice. I’d appreciate if my sim would crash less often and I will, for now, assume that there’s real data behind these graphs, that they were not only produced to have something positive to announce for a change.
My business would be better off though if my customers would receive their wares after paying for them. And if they receive their purchases, which is rarely the case these days, could use them without “item missing from database” errors or missing scripts. I’d rather live with 2 sim crashes per day as it used to be, but have satisfied customers who don’t give me negative word-of-mouth advertising because your asset server is falling to pieces.
April 10th, 2008 at 5:55 PM
its weird but since this new veiwer im crashing 10-20 times per day even if i lower gfx settings all the way down , on the old RC veiwer i could have totaly maxed settings and not crash for a week, now im getting all kinds of bugs memory errors ,constant crashing ,severe loading times i mean its taking me 30 mins to load a sim now… i hope these things will be resolved my sl is almost unusable now, is this just me or is these problems for alot of people?
April 10th, 2008 at 5:55 PM
Sidewinder~
ty for the response. I figured it was reported already. I wonder if it’s the type or script used for rotation because i have one rotation ball that never stops spinning… meaning it’s good. But only that one.
if i can help let me know.
April 10th, 2008 at 5:57 PM
That looks like a very good improvement. Of course, since that took a large portion of sim crashes out of the picture it should be relatively simple now to isolate and fix the other sources of crashes : )
April 10th, 2008 at 5:58 PM
*THANK YOU*THANK YOU*THANK YOU*
I can’t tell you how many times I’ve accidentally crashed my home region because of what I was *certain* was physics collision problems. (objects being rezzed on top of objects in motion… objects in motion crashing into each other, etc).
I’d completely given up on one of my favorite projects because I couldn’t, in good conscience, sell things that I knew would increase the risk of region crashes.
This is a big beautiful *GREEN LIGHT* I’ve been waiting to see for a long time.
*THANK YOU*THANK YOU*THANK YOU*
April 10th, 2008 at 5:59 PM
@32 Ishtara: Which region are you talking about? I’d be happy to check the stats for that region… /Sidewinder
April 10th, 2008 at 6:00 PM
@15: hmmmm… about 15.000 resident regions at the end of Feb, they sell 100 per week (Jack words) so they sold about other 500 until today, then you have to add LL simulators (do they own 1500 regions? …i don’t know, it’s possibile, considering how extended is the governor property) …so Sidewider numbas are almost correct (what news).
April 10th, 2008 at 6:01 PM
Way to go H4 team! Great job! And thanks for being so responsive about the bugs.
April 10th, 2008 at 6:01 PM
Good stats Sidewinder, especially on the physics based crashes. If it’s like this in 6 weeks or so then the proof really will be in the pudding.
Nice to see some useful stats on here.
I don’t suppose you can do anything to deal with the land price crashes can you
April 10th, 2008 at 6:04 PM
Impressive! Now if only client crashes could be reduced by an equal amount *poke*
April 10th, 2008 at 6:05 PM
Back in August at the SLCC I talked to Philip about crashing, telling him I am in game 18 hours a day, and the other 6 hours Im on anti Idle, not loggin out for 5-7 days total. Which means hardly every crashing out of game. Since April 1st. I crash about 10 times a day. Mostly when I open a profile, group or TP. I guess when there are 50% of bots in game running with out a viewer and not interacting with the client, ofcourse there not going to crash. But myself logging in and using SL, I crash way to much now. Most of the time I soft crash, which means I get logged out of SL but I still see the world, I only know Im crashed because the mini map turns red or I get an offline going to my blackberry, which tells me, hmm if Im getting offlines, I must not be logged in to SL anymore.
April 10th, 2008 at 6:05 PM
@30
Sidewinder, you rock! My sim hasn’t crashed since the H4 upgrade went in, and we had frequent crashes before. We keep a crash logger active in the sim, and I monitor the sim stats closely. Not only is the sim more stable, but total frame time is down slightly too since the sim is spending less time calculating physics. Can’t wait for the Mono code to kick in!
April 10th, 2008 at 6:07 PM
This crash reduction is a bunch pf hooey. My sim has been down for the better part of a day. It crashes and then you say problem fixed? I have had prims dupilcating themselves, prims floating in the air and i’m paying a high enough tier and unable to go where I built a house. If this is stable I long for the older setup. This is the second week of outages and all I do is lose money spent on tier.
April 10th, 2008 at 6:08 PM
If I’m reading the charts correctly, non-physics crashes actually increased significantly after this update.
It seems like the average # of daily physics crashes for the seven days leading up to the update was about 1308, and the average # of “all causes” crashes was about 1788, leaving about 480 non-physics crashes per day.
After the update, the average # of daily physics crashes dropped to about 35, and the average # of “all causes” crashes dropped to about 574, leaving about 539 non-physics crashes per day.
That’s a daily average non-physics crash rate increase of about 11% since the Havok 4 update. (All numbers estimated based on the graphs.)
Possible explanations:
1) Previously, it would have crashed for another reason anyway, but it happened to crash on physics first.
2) Some crashes previously attributed to physics are now attributed differently.
3) The physics code is causing problems for some other part of the server code due to bugs or strain caused by increased activity.
4) The rest of the simulator code has become less stable for a wholly unrelated reason.
5) Sunspots.
There may be other possibilities as well. It would be nice to know which it is.
(None of this should detract from the sterling work of the Havok 4 team to increase stability.)
April 10th, 2008 at 6:10 PM
Hi Sidewinder, casual anecdotal evidence in Caledon pretty much confirms your graph, it’s been pretty stable region-wise. Other factors - teleporting and whatnot - well, that’s a different thing and a different story.
We have had a lot of objects de-linked by the changeover; more than I had suspected originally but nothing unmanageable - maybe a few per region of linked objects that were ‘just barely hanging together’ according to the old link rules anyway. The new, seemingly looser link rules has allowed for easy restoration in all cases so far.
Also, the newly prim-doubled openspaces are doing great!
All in all a very good show and a very good transition to better physics.
Desmond Shang
Independent State of Caledon
April 10th, 2008 at 6:11 PM
@45 T N: Yes I noticed that in the data as well, but haven’t yet had the time to research it. I do suspect however, that #1 is the probable cause. There were enough cases that the physics engine would spontaneously crash first that it is reasonable that it was masking other problems that would have showed up had the region survived the physics problem. /Sidewinder
April 10th, 2008 at 6:12 PM
@42….that is weird…… I can run SL happily (with the normal bugs that are present) on an ASUS Eee 4G mini PC that has a 4gb ram drive, 1GB ram and with a 4GB SDHC card as the D Drive. The processor and graphics are below SL recommended standards but it all works fine (running WinXP). I wonder if top spec. PC’s are just to complex
April 10th, 2008 at 6:13 PM
@48 Captain: Just to toss one more thing in the mix, dual displays can cause odd video driver bugs to appear that don’t show in single display rigs, and the bugs found sometimes depend on the particulars on all sides (which port, which display resolution(s) etc… Too many variables
/Sidewinder
April 10th, 2008 at 6:14 PM
all i know is that sl looks better, never crashes on me, does not bog down my pc, and is tight.
i have all my graphics set to the max, even distance and keep high frame rates and it just works really well. =)
thanks Havoc team for doing what must have, at times, been a really daunting task! *ener can’t even hook up a printer* o_O
congrats all of LL
April 10th, 2008 at 6:17 PM
It is good that you are working on lowering the amount of crashes in Second Life. Please keep your software people working on all usability issues including inventory loss and failed transactions. Is there anything that can be done to reduce crashing in crowded locations? Reveal Magazine was promoting an event for Relay for Life and I experienced many crashes during the auction and DJ contest.
April 10th, 2008 at 6:18 PM
No it doesnt crash, it slows down to a crawl with upwards of 40-50 pending downloads. Everyone gives up in disgust because clothing, profiles, etc do not load and then the pending downloads have time to recover.
April 10th, 2008 at 6:23 PM
Can we now see a Cool Graph of the New RC Viewer, to see how many times it has crashed. Assuming that people log their crash report…which they don’t. Cause they figure well it’s pretty much the same cause “I did Something, viewer Crashed it will do it again in a few minutes why log every time”?
April 10th, 2008 at 6:24 PM
@48 Captain maybe LL can change my last name to crash. And Ill be known as Spontaneous Crash instead of Spontaneous Radio.
Just a Spontaneous Thought
April 10th, 2008 at 6:24 PM
Excellent job, Sidewinder and Havok 4 team!
Now, please fix the creeping bug in rotations (objects stop rotating after a day or so) and the unstable hover (avatars sometimes sink, especially after turning).
And my .45 throws a little high and to the left. Could you do something about that, please?
April 10th, 2008 at 6:25 PM
Myself and many other scripters would like to know when you will be addressing the issues of many scripted products Havok 4 has destroyed or crippled . Theres alot of talk in-world you don’t hear about the products that for the most part , just don’t work any more and alot of merchants that made the products have a total nightmare on their hands . What are you going to do to help these folks who worked so hard to make sl a better place only to have their efforts underminded by Havok 4 ???
April 10th, 2008 at 6:26 PM
regarding TargetOmega,..
I have found that only things I have made and compiled before the last update fail to turn/spin.
EVERYTHING I recompiled worked and moved like it should, and continues to spin as I speak. The majority had to do with events activating the TargetOmega function (like touch or linked message for eg) as opposed to having them as a state.
In short, nothing using TargetOmega on the random things I’ve created fail to work, all spin as planned. Only certain scripts where it required an event to trigger the function i found to have issues with.
Great job with h4 btw, it allows me/us to try things that would have been simply too laggy in the old environment (for the most part), shattering linked prims and having them float away at given speeds for eg =)
April 10th, 2008 at 6:26 PM
that’s wonderful dears, considering i just crashed.
April 10th, 2008 at 6:29 PM
I believe Havok 4 will be better somehow but i must be the only one to expreience worse performance than ever. How come i was forced to upgrade from 18.5 to 19.1 and crash more than i have ever before? my old 18.5 never gave me these issues.. are some versions better some peeps and not others ?
April 10th, 2008 at 6:29 PM
Well, I for one can attest to the reduction in crashes (see http://www.iconia.org/sl/crashquery.php for the Isle of Wyrms stats), though the latest sim build seems to have a nasty habit of crashing and staying down for the count. Both Limbo and Cathedral went down today, and the latter required a region down ticket to get running again.
April 10th, 2008 at 6:30 PM
only to find he has crashed typing the above comment
standing still lol..
April 10th, 2008 at 6:33 PM
@56 Hey: We ran a 6 month public beta with hundreds of regions, talked (and are still working with) every product maker who is interested. Had 50 office hours, and are STILL openly inviting anyone with product issues to get in touch so that we can resolve issues… Please make sure that open issues are raised and brought to our attention and we will try to resolve them. /Sidewinder
April 10th, 2008 at 6:35 PM
Wow great job sidewinder & team
BIG THANKS!!! KUDOS!!!
Keep up the great work I hope you will be doing way more projects (and blogposts!)
April 10th, 2008 at 6:36 PM
WTG Sidewinder! And thanks for your update on the siggy swimmer issue, didn’t catch you guys are already working on it. Also thanks for dealing with the open space sim issue so quickly! Your team is def the model that other LL teams should follow, responding to resident questions as they arise and keeping us informed on what’s going on. Now I can just sit back patiently and wait, knowing your team is working on fixing SL swimming
It’s this sort of comunication that people expect and want. We aren’t unreasonable citizens, just need a heads up on what’s going on and why !
Thanks again!
Stephe Ehrler
April 10th, 2008 at 6:40 PM
i crashed so often with the new viewer that i uninstalled it and reinstalled the earlier one. lower crash rates? hardly. my crashes with the new viewer were ten times the number of crashes i had with the earlier one.
April 10th, 2008 at 6:44 PM
As a region owner I have noticed the effects of Havok 4 in a positive way. In both the early adapter and now. My Sim has run stable and as far as I know, has not crashed except for a network issue on the grid. I have restarted it only about 2 times myself since Jan. Great work and it shows.!
April 10th, 2008 at 6:46 PM
Actually, I just checked the times on those. It looks like Cathedral was caught in the “700 down region” issue. Not sure about Limbo, as it was several hours earlier.
Anyway, like I said, definitely a huge thumbs up in the crash-mitigation department. There are a few niggling issues left here and there, but they will likely get mitigated soon.
April 10th, 2008 at 6:50 PM
Crash stats down cuz many of us still cant even login.
April 10th, 2008 at 6:57 PM
I’ve read some pretty ridiculous comments before, but suggesting that H4 undermined scripter’s efforts because of their operable status in an older, less stable platform takes the cake. Things may need to be updated on the object development side instead of some conspiracy theory being suggested.
Anyway…
Sidewinder, oddly enough I’m seeing some previous bugs that were tackled show their ugly heads…
-More than minor sliding on slight grades
-Steady decent while in hover
-Falling while climbing stairs
All the above have jiras for them, but it seems at least these three seem to have got worse recently.
April 10th, 2008 at 7:02 PM
Funny thing is i sit reading this while awaiting the Grid to stop crashing
Way to go Havok team!!!!1
April 10th, 2008 at 7:02 PM
Ok, with region crash, wonderfull,
but if i only see me, i got crash at moment, and it was the 6th in last 8 hours, so, much more then some weeks before.
I thnk, my crash rate is going higher, but its only me ?
ty.
April 10th, 2008 at 7:04 PM
Hooray for less physics crashes!
Boo for my sim being unable to send outgoing IMs and objects, including SLX deliveries, for six hours until a concierge moved us to a new physical server!
Sorry, guys, but I’m not cheering until one day goes by where I don’t have some transaction/asset server failure, sim crash, communications problem, or whatnot.
April 10th, 2008 at 7:04 PM
Again: “Lower crashes” means lower number of *SIM* crashes, people. As in, “The simulator you are in is currently going down, you’ve been disconnected, press continue to still view chat history…”
NOT “I rezzed an object / tp’d / looked at a prim sideways and crashed.” That’s a *VIEWER* crash, not a *SIM* crash.
April 10th, 2008 at 7:06 PM
@ almost 50% of the previous commenters:
sidewinder is talking about SIM crashes… that is the part of SL that runs at LINDEN LABS. or in other words, not on YOUR computer.
April 10th, 2008 at 7:07 PM
i am getting the descent when hovering again and the falling on stairs sometimes.
but .. we were in on the early adaptor.. we never crashed once (well not the SIM.. we were offline when the whole world was too) and haven’t since.
there are some things that dont work on Havok4 but its a small price to pay for SIM stability…
Sidewinder and team deserve the cudos…
also for great communication.
the rest of the borked stuff… still tres borked as is the communication thereof.
April 10th, 2008 at 7:08 PM
Very well done Sidewinder and your team !!
Renee Faulds
April 10th, 2008 at 7:08 PM
@sidewinder: what about almost anything that has physics enabled and interacts with linden water (as in, boats, surfboards, scuba gear, or swim attachments) not working properly? that problem kills about 75% of my favorite SL pastimes…
April 10th, 2008 at 7:08 PM
I’m happy with the reduction in simulator crashes. Way to go!
Now… since I updated to the new RC last night, I CLIENT crash once every 20 minutes. This is new and frustrating. Could someone e-mail me and tell me if there’s a temporary workaround for this? It seems intermittent and unpredictable. I really don’t want to have to roll back to the old client just to get my work done.
Please please e-mail me if there’s a workaround. This is a showstopper for me.
April 10th, 2008 at 7:08 PM
To those that think these stats are fake, I can assure you they’re not fake at all.
My sim of Hypnos was crashing 2-3 time *a day* before I enrolled it in the Havok 4 Early Adopter program. From the moment Havok 4 was put on it, until now (a 6 month period), we have had a total of 2 crashes. And both of those were attributed to bugs we found and reported to the Havok 4 team.
The sim is a high traffic sim, with a constant population of around 20 avatars there. I suspected once it was rolled out grid wide, similarly huge increases in stability would be reported, I’m happy to see it was so.
April 10th, 2008 at 7:12 PM
Nice job, Sidewinder ! It’s great to fly over a sim boundary without going nuts!
April 10th, 2008 at 7:15 PM
Well after being overjoyed at how the Windlight Release candidates have run over the last months I’m quite disappointed in the crash rate of 1.20. I’ve logged countless hours in game in pre-1.20 and MAYBE crashed 3 times I can remember. Since I’ve upgraded yesterday to 1.20…I’m crashing about 3-4 times per hour and its starting to get a bit frustrating to say the least. I’ve done the usual clear cache, reduce the viewer to minimums, etc (shame cause I’ve been running at highest quality these last few months) to no avail.
Also, it seems the preferences for whether I want to run in a window or full screen are not being saved after modifying that option - both pre logon and post logon.
April 10th, 2008 at 7:17 PM
Thankyou! I have noticed less sim crashes. It really does help
Way to go.
*does a backflip to prove it * thankyou for your speedy help.
And Im doing well with adjusting to the changed camera thing
April 10th, 2008 at 7:18 PM
Naoki, you can still download the old viewer and go back to that one till its fixed
A big thanks for Sidewinder and the team you did an amazing job!
April 10th, 2008 at 7:22 PM
Sidewinder, I really appreciate the communication you’ve brought to the blog and the dilligent effort you and your team are making to improve SL. I have a studio full of Mac’s. I am uncertain which part of all the improvements has specifically ruined SL for me. I can’t tell if the server is better or worse because the forced viewer upgrade has been hell across our studio. I’ve had support tickets in since March 13 about the viewer and only had one response about it saying it should have been fixed by all the new software, I’d agree, it should have been but it’s not. If I tp out of a region, I’m logged out. If I walk across a region border, I’m logged out. My “About Land” is slow as mud. 25% of the time a notecard does not save and says there’s a problem with the server, try again later. I can’t add people to my group most of the time and this has made managing my properties nearly impossible. Many of my other developer friends are as disappointed as I am with performance. After we’ve worked for a year to build our businesses, we’re watching them be torn apart by performance issues, customers running away pissed off because vendors are so slow or don’t work at all. I only wish that another Linden would step forward and let us know wtf is going on and when there might be some stability.
April 10th, 2008 at 7:23 PM
I can make graphs on my paint program to to say whatever I feel like. I am one of those that you are crashing more and more every day. Please stop adding more stuff and fix what you have.
April 10th, 2008 at 7:24 PM
@84 Bud: Please IM me inworld so I can understand these issues better… Many or most of them seem to not be simulator nor Mac-specific, but I am curious to get a better handle on the specifics of your situation. I am not promising that I can fix or even improve it, but have a couple of theories about one of the behaviors that I’d like to check out… /Sidewinder
April 10th, 2008 at 7:26 PM
@85 HonkyTonk: These graphs were not made up. If you have crashing in situations where not everyone on the region is crashing at the same time, then you are experiencing a viewer crash - the type where the software on your computer crashes. The crashes that we have reduced with this project are on the server-side, and affect all people with avatars on a region at the same time. /Sidewinder
April 10th, 2008 at 7:29 PM
Funny how people read things differently isn’t it?
Sidewinder, congratulations to you and your team for a great job.
Hell my release candidate viewer, is as crash prone as any other “Winslows” application. I’ve learned to live with a relog from time to time, sorry people, read the blogs properly and get off sidewinders back.
There is a viewer development team as well you know.
And a whole mess of folks who are working on the SL Open source grids as well.
Thank you Lindens, for opening your server and viewer source to us who like to tinker.
I for one love this system and will defend it and the teams of very hard working people who have put it together for our enjoyment.
So it is a little buggy, what isn’t? MS Windows? Hah!
That is what research and development is all about.
Learn to live with it, and love it.
Again, Sidewinder, thanks for a damn tough job, well done.
April 10th, 2008 at 7:46 PM
@55 Lindal Kidd……. LOL (.45)
@58 webmasteramadeus
*********This is about the SIMULATOR not your client crashes.
65 bret barbasz
*********This is about the SIMULATOR not your client crashes.
85 HonkyTonkAngel Weatherwax
*********This is about the SIMULATOR not your client crashes.
What is the deal with ppl not differentiating Sim/Server from client
April 10th, 2008 at 7:47 PM
88:”Thank you Lindens, for opening your server and viewer source to us who like to tinker.”
-
….when did they opened the server?
April 10th, 2008 at 7:48 PM
excellent work.. havok4 is great! if only the windlight team and the user interface team could provide a more stable viewer to enjoy the new havok
April 10th, 2008 at 7:49 PM
@MimIi: I said, “I really don’t want to have to roll back to the old client just to get my work done.” Still, fewer sim crashes means going back to the old client isn’t such a bad idea after all. I guess I will.
April 10th, 2008 at 7:58 PM
Sidewinder, my sim is running perfect since I bought it last year in august, with or without HAVOC4. Guess that the biggest problem for simcrashes are caused due to complex scripts, megaprims, etc … on those sims !
despite all the good work from your dept. which we should encourage ofcourse, it is a fact that SL is experiencing more and more problems, I follow this up on a daily base, I meet and speak to loads of people around the grid, trendwise I dont need to make any graph to the crowd here, it is a fact, the sl fun experience is going backwards !
SECOND LIFE is promoted as the world that ‘we users’ can create ! right ? So that means you launch a product that is working at all time and the users create there worlds at all times !
facts :
It is impossible in rl in any business to launch a product that is still in testphasis and being tuned or upgraded every 2-3 hours (and I know what I talk about) ….
*car manufacturers will not launch a new car when the engine is still under test developement ! Would they recall the cars worldwide back into the service dept every week ? I guess not !
*a barebone system running in a company where 10.000 users are connected to it worldwide (24/7). Can they permit to see there barebone being upgraded every 2-3 hours causing the whole company being down for hours or constant interruptions ? NO …. forget it !
sidewinder, as I already mentioned in other blogs, there is no direction anymore … people just need a SIMPLE STABLE VIEWER that let them CREATE there things in SECOND LIFE or INTERACT with other people and a viewer who is accessible for most people allround the world, cause with 50.000-60.000 avatars connected average in which 60 % are camping for free money and the other 35 % loosing business (people who invested and who were working very hard in-world to compensate parts of their tiers !) …. I am not much impressed … most sims are empty, abandonned or are looking like a dump !
LINDEN is now focussing on improving the VISUAL QUALITY, I will tell ya straight away that ya will never have the money and the payback to reach a profitable goal for this, cause guess LINDEN LABS is not a charity organisation ? People need to be paid, operational costs, etc …. Do you really think that second life users are going to upgrade there pc’s moving forward to next example like my personal pc (INTEL 6600/4GB MEM/1TB HDD/8800 GTS 512MB VGA …. ) ? If ya want VISUAL QUALITY then people need to stop using SL and move to the REAL GAMING WORLD where the gaming industry sells like MILLION cd’s in different platforms XBOX, PS2/3, PC @ 60 US$ per game ! Compairing this with the approach from linden makes me think about the consequences of your actual strategy !
anyhow, I see/feel more and more that the fun is more than over, seeing that I am part of A WORLD where I pay 295 US$ per month and paid 1695 US$ for a sim, knowing that the product is still not finished and being tuned every hour of the day ! like ; Will my car start today or not , same comparaison?
again thx for your hard work sidewinder and your team, but youre only a small fraction of the whole issue, a team is a whole company with a clear vison and strategy, it also doesnt help then to give the customers the impression or false hope in BLOGS like here that it is going in the right direction, please dont do that !
I hope that the new CEO will have a strategy and a strategy he can execute without being influented or interfered by the board every second.
April 10th, 2008 at 8:02 PM
@93 Cosa:
I am not trying to “give the customers the impression or false hope in BLOGS like here that it is going in the right direction”.
I am reporting on the results of the Havok4 team, in objective factual terms to demonstrate that this project has delivered tangible results that improve stability of the Second Life platform - our stated goal when we announced the beta process for this project. If demonstrating that we delivered on the stated goal of this project is problematic, then I surely don’t know what I should have said instead once the statistics were in hand!
Regards,
Sidewinder
April 10th, 2008 at 8:04 PM
Apparently no one has been paying much attention then… last night ALONE, I surpassed this “estimate” with the new viewer.
The “normal” viewer works now.. with its “normal” crashing.. but, the newest viewer.. OMG… please do not make it apart of the main..
It’s insane.. we had three other ppl online, we thought it could be the sim, so we moved.. NOPE.
And its still not working right today.
For a program that is CLAIMING lower crashes… I have to say.. the data is WAY wrong..
Where are you testing?
And please, do not try the “its your machine” crap… because its not.
I surpass the req machine.. so.. that is NOT the issue.
April 10th, 2008 at 8:06 PM
@95 Morgana: This is server side crashes, not viewer crashes. Please see initial post and comments above. /Sidewinder
April 10th, 2008 at 8:08 PM
Well, Congrats, but , please .. fix the problem about the physical objects with the Havok, the elvators, and other stuffs don’t works correctly …
Thanks, Pred
April 10th, 2008 at 8:09 PM
Hats off to the team.. Every step you make to improve the SL experience and stability is appreciated!!
Pat yourselves on the back and look for the next dragon to slay.
April 10th, 2008 at 8:10 PM
Perhaps a lot of the people complaining about viewer crashes should take their head out of their … and actually check out their PC!
Yes I do crash sometimes, but its not unusual for me to be in world for 18+ hours at a time with no crash. My PC isn’t great (2.4Ghz, 1.25 Gb Ram, old video card and I run two monitors)
You know maybe your setup is borked, bad/old drivers, windows crapped out etc.
Not everything is LLabs fault you know!
p.s. Congrats Sidewinder & Team, and I have a bug for you… I can’t drive my car on the road outside my house….errr hold that thought couldn’t do it in H1 either….there is no paving! (can we put you in charge of the road building crew?)
April 10th, 2008 at 8:14 PM
@95 Morgana Hilra
They chart is about Server Software Crash Rate!!! The Sim’s
Not your client crashing.
***We had a big meeting today in Pooley about the client crashing, They know its crashing thats why they held the town meeting.
Server crash rates, everybody say that with me…..
Hello, can anybody hear me, is this thing on…
April 10th, 2008 at 8:17 PM
@ attempts to spin this positively notwithstanding: the development of the client took a seriously wrong turn about two versions ago. Featurities. The current viewers are so overloaded with instable features that even running them on minimal settings takes a heavier toll on your system (because of all the internal filtering necessary now, persistent memory leaks, and a bigger “footprint” of the UI itself) than running previous viewers on maximum settings does.
As nice as it is for us sim-owners to see server-stability improve (and thus reduction in content-loss), it doesn’t mean much for the average user or potential newcomer.
I’m glad this issue is addressed, but the way things are looking the degradation of user-experience due to an incompetent client-design and its featurities will more than make up for the gain in reliability of the servers.
April 10th, 2008 at 8:20 PM
Poor Sidewinder, having to answer the same questions over and over from people who can’t tell their butt from their elbow.
April 10th, 2008 at 8:21 PM
Excellent news. The results are very visible in-world and a great improvement. Thanks to the Havok 4 team for all their hard work and thanks for the good updates, I wish all teams at LL could keep as interactive with the populous.
I’m really looking forward to seeing the Mono team’s product going live as well and would like to hear more from them as that nears production status.
April 10th, 2008 at 8:23 PM
If half of you all have even been part of the Havok4 testing and have attend the meetings you would have an idea of whats going on.
Also viewer crashes have nothing to do with Islands/sims and is not part of Havok4. The crashes reffered to her are sim/island crashes. Not viewer crashes.
You need to complain to the Viewer team about viewier crashes.
April 10th, 2008 at 8:25 PM
Sidewinder - you and your team rock. Hopefully every project team at LL is paying attention to how you operate.
April 10th, 2008 at 8:25 PM
Great, lower sim crashes is always good news…can anyone explain to me why my screen keeps flashing intermittently, or why the v.1.20 seems to have removed map rights? Nitpicky perhaps, but little oddities that were definitely NOT issues before…seems we’re playing whack-a-mole with bugs and issues…smack one down and three new ones pop up!
April 10th, 2008 at 8:27 PM
@101 ChatNoir Moonsoo
LOL so now you are saying that fixing the sim crash rates somehow in some wierd way will give the new user a bad experience or somehow because the client is having problems that they shouldn’t make the sim better.
Wow that’s a piece of logic there.
April 10th, 2008 at 8:28 PM
Try researching stuff before posting, and not even having a clue what you all are posting about.
Actually try attending one of the havok4 meetings. Get a clue before posting so you all don’t look like fools
April 10th, 2008 at 8:29 PM
Holy crap, Sidewinder!!
I hope you and your team are getting tasty bonuses for this.
Congrats!