Cory Linden Town Hall
Wednesday, May 2nd, 2007 at 4:41 PM by: corylindenI will hold a Town Hall tomorrow to discuss the stability and performance issues raised in both Project Open Letter and additional questions posted in the feedback forum. In this post, I cover some of the questions raised in the letter, but first two comments.
First, thank you for the effort you put into Project Open Letter. We are working hard to maintain effective channels for communication and from a developer’s perspective, a clear listing of specific problems passed along so politely is much appreciated.
Second, I am sorry for the problems you are experiencing. I understand the time and energy you are putting into Second Life and am making every effort to ensure that Second Life is what you need it to be.
Inventory loss
Inventory and other agent specific data is distributed across multiple MySQL databases we refer to as inventory servers. The name is historical since a lot of data beyond inventory is stored there. This partitioning scheme provides scalability for much of the agent data but currently relies on manual rebalancing after periods of heavy Second Life growth. The scripts that drive rebalancing, while heavily tested and in use for over a year at this point, had a serious bug related to error handling. This resulted in inventories potentially being damaged during the transfer. The good news is that bug has been fixed and new transfer scripts deployed.
There are other errors that could manifest themselves as inventory loss, such as dataserver stalls, failure to complete transactions correctly, failure to properly cache the inventory on the client, or failure of the asset system to deliver an asset pointed to by inventory. Two separate teams are examining these problems, and there are numerous fixes either recently rolled out or in QA. The first team is focused on fixing existing bugs, while the second team is developing a more robust, distributed transaction system. We are also reviewing the design with outside developers familiar with building transaction systems. As we get farther into the design process on the next generation transaction system, we will publish the spec as well.
Inventory limits
We do not have any projects underway to limit inventory sizes, although it has been discussed. We have been doing a large amount of data collection in order to make better decisions about where to set limits, predict equipment purchases, and to understand inventory use. Also, note inventory size is not a source of the problems you have been seeing, except in a second order way because large inventories cause us to purchase inventory servers a bit faster. As such, errors such as the previously discussed transfer script error will impact slightly more people when we rebalance after new equipment is installed.
Inventory backup/local storage
No current project on this either, although most of the groundwork has been laid to allow it, so we could bump the priority for Q3. The design issues here are generally non-technical in nature and are instead related to metadata preservation and permissions. Since backups and local storage are effectively making extra copies of assets, they are a real opportunity to leverage Creative Commons and other licensing schemes that support copying.
Friend lists
The in-world friend list was broken with the 1.15 release. Simulator level caching of friend data was accidentally removed, resulting in a large increase in load on backbone. This overloaded backbone and agent presence stopped functioning. While debugging the problem, the web friend list was deactivated in order to reduce some of the load. Although some of the backbone problems were fixed last week, a small API change in those fixes again broke the web friend list. That was finally fixed with the web push yesterday.
Despite all those fixes, it appears that a bug still remains whereby the viewer does not receive an accurate friend state snapshot on login. Since the presence information is cached and only updates on changes, this means that until friends log in or out, the information on the viewer could be out of date. We are chasing that bug and any additional information or repeatable test cases would greatly help, so please head over to the Public Jira if you have anything to add.
In the long run, presence is another project that we need to build out in a far more scalable manor. A core part of our next generation design is presence, and like transactions, we are currently reviewing our designs with external experts who have built related systems.
Find
We have an intermittent crash in MySQL related to find queries. As a result, we have had irregular Find outages due to this crash. We are chasing the problem but have not been able to solve it yet, since the query in question is often issued safely. We are also in the midst of an upgrade to MySQL 5.0 as part of overall infrastructure upgrades. Since 5.0 has superior debugging information, we expect that even if we haven’t found the problem before the upgrade, we’ll have more information about it afterwards.
Moreover, improving in-world search is a major Q2 project for us. We feel this is important not only because it will improve the user experience for everyone but because search is currently hitting a central MySQL database, so as part of our broad improvements to search, we are also building out a more scalable solution. More details on search will be forthcoming soon.
Grid stability and performance
As an aside, problems like teleport failures and inventory issues are not related to either Havok or Mono. While both will bring improvements to individual sim node’s performance and stability, they have no appreciable impact on problems related to back end systems. Havok 4 is in testing prior to hitting the Beta grid and the Mono project has fixed the major blockers for us, so we are waiting for resources to free up from other projects there.
Teleport failures could be the results of many different problems, and are definitely exacerbated by problems in agent presence. We have a team currently investigating this problem. Again, additional data points and reproducible cases would help them a lot.
Build tools
Our studio focused on live grid problems is taking a look at reproducing these. Obviously the build tools are critical to content creation so hopefully we should be able to get these fixed quickly.
In terms of overall development effort, four of our internal studios are focused on issues in the letter. Currently, of the 54 people in development, program management, QA, and web development, 37 are directly working on these problems, or 69%. We have 5 developers hired who have not started yet, and all of them will go onto these bugs initially, raising it to 42 out of 59, or 72%. Since not all tasks are equally suited to all developers and additional projects need to move forward, I feel this a very appropriate level of attention.
I hope this helped answer at least some of your questions and I look forward to talking about them more tomorrow.
May 2nd, 2007 at 4:45 PM
[...] Second Life Video Tutorial Series (beta): Intro to Textures Cory Linden Town Hall [...]
May 2nd, 2007 at 4:53 PM
Thanks Cory. This run down covers a good many of the problems that people have been complaining about, and I’m sure it’s going to go a long way towards damage control for tomorrow’s town hall (may it be peaceful and productive!).
I’m very happy to hear about both Havok and Mono being well into development and at least within sight of the beta grid, if not waiting at the gates to be released into the wilds. I’ve still got a whole slew of Mono related questions to go through, but the first two seem to have been answered. I’d still appreciate a Q# estimate for beta grid trials of Mono though. >
May 2nd, 2007 at 4:54 PM
I think inventory limits are a very poor idea. What will we do if you bring them in? We’ll have to delete some of our items, of which we paid our own linden for.
Will we be reimbursed? We should be if the limit happens. Because it’s not our choice.
Honestly, I love my vehicles, I love my pets. I couldn’t see not having them. I own tons of suits. Imagine if I had to limit my choices more.
I couldn’t shop much with a limit placed. In time, that would impact the SL economy in a very negative way. Folks would buy less. They’d then have to worry alot about what stays/goes.
May 2nd, 2007 at 4:59 PM
Can someone also look into the draw order problem that the last update caused. I realise this change was an attempt to fix a problem with the Linden trees, however the impact on the main grid should have been considered before it was implimented. This problem rendered over 90% of my flexi hair unusable and all my non linden trees look like….”shrugs”…. try looking for yourself.
May 2nd, 2007 at 5:07 PM
Thank you thank you for coming forward with some solid information. This is something that would be really good to have on the blog as a regular thing.
That it has come as a result of a last-ditch effort by residents to get some response is a bit of a shame - but it’s here now.
So please - KEEP US IN THE LOOP. We’ll all feel much more like we’re not being ignored.
May 2nd, 2007 at 5:18 PM
Thank you so much for your information, and I am looking forward to the town hall. I think you guys are doing a fantastic job with so few resources. LL’s recent response to the open letter and the upcoming town hall are good steps forward. Now I feel like LL cares, and before I did not.
May 2nd, 2007 at 5:19 PM
this is just the sort of communication that the letter, unfortunately, had to spur. please try to keep it up. make it part of your job description to inform us on an ongoing basis as to the technical issues the lab must deal with. we all realize the immense chores related to maintaining the stability of all the game features and/but we don’t expect you to stand still either.
May 2nd, 2007 at 5:20 PM
Why is MySQL used instead of full SQL? There seems to be so many limitations with MySQL that I can’t imagine that it is the best database for the job.
May 2nd, 2007 at 5:20 PM
Regarding grid stability, it is not uncommon to be the only avatar in a region that is otherwise script/activity stable, wearing a lag monitor, and see 2-5 second lag spikes as high as 10-15% and even higher. To what should these occurrences be attributed if not uneven load-share within or across the grid?
May 2nd, 2007 at 5:23 PM
Thank you Cory, great information, and yes it will go a long way to reducing the complaints tomorrow - answering people is easy after all rather than ignoring them.
However core issues remain, like lag, logins, money loss and the fact that every upgrade seems to bring more problems than it fixes, indicating insufficient testing.
A good example is the sever update just issued. It’s supposed to have fixed some of those Group IM issues, but it certainly hasn’t for me. My first message got through okay, every other one has failed.
However, two additional problems have surfaced, one was notecard links to textures returning completely different textures that were not even mine even after three relogs (although since I was logged out by the system and moved to a different region, that has been resolved), and the fact that I now have HUD attachments that have been taken off and won’t wear back.
So that’s a frustrating example of the “fix” not having fixed what it was supposed to and creating yet more problems.
And a big issue to address is the effect policy is having on business in SL (that is proper business as opposed to property reselling). Peak users used to follow total users at about 1%, but that has now been static at 38k or so for two months. In that time, there are now more times during the week where that limit is reached, and that is when all the problems occurr : tp failures, money missing, inventory loss etc. That sounds like system overload not bugs.
However at the same time, available land has pushed out at 20% per month, meaning that things are effectively more spread out now, and average number of avs per sim even at peak has fallen to less than 5. If land gets out of sync with users allowed in, then that hits all businesses.
May 2nd, 2007 at 5:31 PM
Someone talked to me!
May 2nd, 2007 at 5:31 PM
I have had a serious crashing problem since the upgrade. I had the same problem on the last upgrade and went back to 1.13 to fix it. Now with 1. 15, the problem is back. I can hardly stay logged of for 5 minutes without everything locking up on my computer requiring a complete restart. I have filed bug reports, crash reports, email reports, and even tried to call. I have recieved some email reports directing me to the website and I have tried all the things on there. With no live help inworld, I don’t know where to turn. I know all the problems are important, but what good is any of it if we can’t stay logged in?
May 2nd, 2007 at 5:31 PM
Was needed ONE OPEN LETTER for your answer…
THANKS LINDEN LABS, very thanks
May 2nd, 2007 at 5:34 PM
I would like to address something that just happened today, which is a coincidence because it’s never happened before, and I believe is a direct result of this petition many of us have signed. I just received 2 emails from someone at LL about bug reports I filed way back in version 1.13, and asked me that if I was still experiencing the problem to submit NEW bug reports under the 1.15 client! WTF?
Why weren’t these read before now? How many bug reports do you guys ignore???
May 2nd, 2007 at 5:35 PM
I’m not crazy about the idea of inventory limits. As a budding creator, I have a lot of stuff in my inventory that I’m working on and I don’t relish the idea of having a cap on the number of things I can make.
Maybe if inventory limits were necessary, they could be placed on NPIOFs, as another way to motivate them to upgrade, the way certain websites allow you more features when you pay for them.
The fact is, Inventory eventually becomes self-limiting, as people get sick of keeping track of so many things and start getting rid of stuff they don’t use.
May 2nd, 2007 at 5:35 PM
Thank you for the information. This would be a great weekly feature. This is the type of information we would like to hear from SL. As Premium members of Second Life, I feel that we should all be “in the loop”. Lately days go by and we hear nothing from SL but spontaneous chatter about new features on the blog. Please keep messages like today’s coming. We deserve at least that much for investing time and money into Second Life.
May 2nd, 2007 at 5:36 PM
Woo! Thanks Cory! It’s nice to just hear what’s going on. Any info on why SL keeps crashing for users after the 1.15?
May 2nd, 2007 at 5:39 PM
I have a serious question for the commentors..
Have you ever seen the Beta grid at all? I was off in the Voice Beta last night and was very saddened to see less than 20 people there.
I feel you have no right to complain about poor testing if you are unwilling to even try the beta grids and first-look clients.
This is only My Humble Opinon.
May 2nd, 2007 at 5:46 PM
I really dont know much about all this, but on the asset or inventory , transaction thing, couldnt people maintain all inventory ? Like we all have thousands of item in our inventory. Rarely do we use a fraction of the inventory that we do have. So not knowing what i talking about couldnt we do something like AOL wher they compact or zip unused inventory till its needed? thereby freeing the servers up for more inventory. Sorry i dont know why i wrote this…
May 2nd, 2007 at 5:50 PM
So will this be just patching and wishing or will theis be done right this time. there are many “IF(S)” in Cory remarks here. What with the “Simulator level caching of friend data was accidentally removed, resulting in a large increase in load on backbone. “YOUR JOKING RIGHT?! How can happen? And where are the heads of the coders when doing this? I don`t blame steve linden onthis at all. But those otehrs coders really need to learn before even trying to do this type of work. this is unforgiveable PERIOD!
May 2nd, 2007 at 5:55 PM
PLEASE PLEASE PLEASE CONSIDER LOOKING INTO THESE OPTIONS.
PLEASE PLEASE PLEASE CONSIDER LOOKING INTO THESE OPTIONS.
PLEASE PLEASE PLEASE CONSIDER LOOKING INTO THESE OPTIONS.
A couple town halls ago, I spoke with phillip about RAM based SAN systems for your databases — Specifically, TMS’s RAMSAN-400 based TERA-RAMSAN.
A simple way to improve asset retrival and storage would not be to impliment a backup system but to impliment an ARCHIVE system, especally combined with something like the TERA-RAMSAN.
It’s quite simple — Impliment the primary databases onto the TERA-RAMSAN cluster, for current inventory items. When inventory items are NOT USED for 3 months or greater, dump them out to your current HARDDISK based storage cluster.
Impliment another tab in the inventory like Recent Items. Name it Archived items. Duplicate the folder structure of the All items tab.
Start moving unused items into the Archives. Leave used items in the All Items tab. Cache the entire inventory directory listing.
Have Inventory Search search the cached listing. Show Archived items as ITALIC.
When archiving items, clear out duplicates. For instance, thousands of newbies have the exact same boxes of freebies. These only need to be stored ONCE, unless they’ve changed.
And here’s another tip… You have a massive cluster of machines that are SITTING MOSTLY IDLE.
Start playing around with Xen on the beta grid or Uma grid.
Xen can migrate an whole virtual container from Hardware Server to Hardware Server in somewhere between 100ms and 500ms.
Start by balancing the load on your grid.
Impliment SIMs as Xen Containers.
When a SIM is empty, migrate the container to a FULL hardware node.
Since physics simulations and scripts don’t cause much load, you can pack MANY empty sims on a single hardware server.
This leaves many more nodes free to run simulators that HAVE MANY PEOPLE in them THAT NEED MORE CPU TIME.
Xen can rebalance your grid and load in REALTIME.
You can also bring new simulators online without having to purchase new hardware immediately, as they are now DECOUPLED from the hardware. This ALSO means you can rebalance the other grid services nessicary to keep more than 40,000 people online at once. When the SEARCH system goes down, Xen can fire two more up immediately, and allow you to do forensics on the container that went down, without having to resort to immediately restarting it to bring the service back up because people are complaining.
http://www.xensource.com/products/xen/
http://www.superssd.com/products/tera-ramsan/
PLEASE PLEASE PLEASE CONSIDER LOOKING INTO THESE OPTIONS.
PLEASE PLEASE PLEASE CONSIDER LOOKING INTO THESE OPTIONS.
PLEASE PLEASE PLEASE CONSIDER LOOKING INTO THESE OPTIONS.
May 2nd, 2007 at 6:01 PM
PLEASE consider hiring Kamilion, he seems to have a more solid plan than I’ve heard from any Linden in a very long time!
May 2nd, 2007 at 6:03 PM
@ 14 ” CyFishy Traveler Says:
May 2nd, 2007 at 5:35 PM PDT
I’m not crazy about the idea of inventory limits. As a budding creator, I have a lot of stuff in my inventory that I’m working on and I don’t relish the idea of having a cap on the number of things I can make.”
Maybe if inventory limits were necessary, they could be placed on NPIOFs, as another way to motivate them to upgrade, the way certain websites allow you more features when you pay for them.
The fact is, Inventory eventually becomes self-limiting, as people get sick of keeping track of so many things and start getting rid of stuff they don’t use.”
Well i stop shopping then if i have to limit my inventory. No money spent means no money back in the ECON for LLABS. ITS THAT SIMPLE!
WHY WHY WHY doesnt LLABS JUST BUY MORE HARD HARDWARE! You Want members right? your letting in all these non paying account thats add only 3% if that to the overd all econ of SL ( IF THAT)?
I tell you what LLABS I live in tokyo and i can get many great deal on hardware If you need help buying from here I can help……:/ If price is the issue here…..Gesh I understand this type of platform is new land we are breaking. But nothing works IN THE RL BUSINESS WORLD if you are just PLAN CHEAP TO START WITH! With all the so called highly educated staff you have in your HQ. Nobody can figure out the correct way to deal with aging asset servers? Or figure out a way to buy new ones…….I all do repect ( and i do have it for LLABS ) Sooner of later that asset servers will do a 80% failure and that wil be enough to cause all most al of us to lose everything we worked for over the years here. That we don`t want happening.
Usagi ( Thank you for reading )
May 2nd, 2007 at 6:06 PM
[...] Linden has written a post on the Official Linden Blog regarding the upcoming Town Hall Meeting and Project Open [...]
May 2nd, 2007 at 6:06 PM
Why not make a place where people with ideas, like Kamilion can post them? Im sure the lindens think about these issues too (more than you will read here probably) … but having some people think along would not hurt I guess!
May 2nd, 2007 at 6:07 PM
@17 - Wasn’t voice beta somewhat private? Correct me if I’m wrong. I know I’m not intending to be a part of that beta for a number of other reasons, mostly cuz really, voice dun fit me.
@Cory - This is a step in the right direction. Thankoo. While I am glad to hear where a lot of these issues stand, what I felt was missing was a sense that these are indeed takin’ precedence over some of the upcoming features. I know they’re different teams an all, but it seemed like a lot of the frustration was new features coming before some very necessary fixes.
May 2nd, 2007 at 6:10 PM
First, thanks Cory for all that information, it helps a lot to have an idea of what’s going on.
That brings me to my first comment, and its not new, but I think needs to be emphasized. The communication between the Lindens and their customer base seems to have dried up - and really seems to have done so as part of a strategy, with so many different forms of communication disappearing. This is extremely frustrating, and certainly not how ANY business should treat its customer base. A great example is the recent turning off of the website friends list. Cory, you’ve now explained nicely what happened and why, but until your blog, there had been deafening silence. Apart from the cryptic notice on the site that the friends list “has been temporarily disabled”, there was absolutely ZERO information given out. Were we to think that “temporarily disabled” meant for a day (well, obviously not), a week, a month, or somewhat indefinitely? COMMUNICATE with us, Lindens! Its critical, and if we had even a small understanding (such as Cory has given here) of WHY something is broke, WHAT is being done to try to fix it and WHEN we might possibly expect a fix, it would go a long ways.
Secondly, again for emphasis, there’s sure seems to be WAY too little testing being done before these frequent releases are thrown out to the general user base. As has been pointed out, today’s update is a wonderful example of that. With the lack of fixes for most (not all) of the group IM bugs, along with the introduction of new ones (closing a group IM doesn’t close it beyond when the next line shows up, at which point it instantly re-opens - over and over and over again) its hard to imagine that even a LITTLE BIT of testing wouldn’t have shown those bugs before releasing it upon the main grid.
Finally, I agree with Ty Voss’s comments/questions about the useage of MySQL for the SecondLife database systems. It seems like MANY of the issues and bugs relate back in one way or another to the database system. If what I’ve read in the forums (which is already out of date) about over 2.5 Petrabytes of data, than that’s more and larger databases than the majority of the Fortune 500 runs. And you’ll find a vanishingly small percentage of such firms trying to run their businesses on an open source DBMS such as MySQL. I’ve been in IT and IT management for over 30 years now (yeah, I’m one of the older SL’ers) and IMHO its FAR past the point where LindenLab should be looking at an enterprise class RDBMS system like Oracle or DB2 to run their gigantic distributed databases on. Yes, it will cost money vs being free - but those enterprise-class systems are without a doubt a much more robust system than MySQL - and again, it seems like a very large percentage of bugs and issues in here ultimately lead back to the database systems and their obvious lack of robustness.
Please take the above as calm, hopefully useful comments, and not as Linden-bashing. Thanks Cory for the information, PLEASE continue to communicate with your customers!!!
May 2nd, 2007 at 6:10 PM
I dont want inventory limits NO WAY
ive only been in SL for 6 months, and i purchased an island with buildings and a history. i have 17K items in my inventory.
i could get that way down if i dumped my trash, but 17K is my life now.
i bet you could help us reduce our inventory without using limits by adding a few features that would prevent us from picking up in a million of the same tree, or making us name all the things called object.
May 2nd, 2007 at 6:12 PM
http://secondlife.com/community/bhear.php
The voice client is there for anyone to download and play with.
I wasn’t too excited about it till I started playing with it. Now I can’t wait.
Oh, I guess I should chime in with {B>Yea! Feedback from Lindens! Keep it up.
May 2nd, 2007 at 6:14 PM
That was…
Yea! Feedback from Lindens! Keep it up.
Danged inability to edit posts.
May 2nd, 2007 at 6:16 PM
i realize that we cannot have a storage backup for items we do not create cus of the permissions but what about allowing us to back up our own creations we make in world that would free up alot of peoples inventory spaces and make things easier to organize we need better search tools in our inventories like being able to search by creator or description SL has bugs that revert named objects back to object or new objects its very annoying to have a 1000 items named object in your inventory after you have named them please think about this allow us to save our own creations at least. also some way to reload our inventories would be nice so we dont have to relog when they get stuck loading in a high lagged area
May 2nd, 2007 at 6:27 PM
This is the sort of communication that has been promised to us time and again yet has never materialized. Why did it take hitting you between the eyes with a two-by-four before you deemed it appropriate to give us this information? If we had been kept informed like this on a regular basis you would find a lot less angry customers.
You have given us information on how many people have been tasked with fixing bugs and dealing with the various issues addressed in the open letter, how about a breakdown of how many people have been tasked with improving the abysmal customer service that has plagued us for so long?
May 2nd, 2007 at 6:31 PM
For a quick solution for Inventory Backup, I personally and I think many others would be extremely happy if for a start we could at least backup objects/items that have ourselves as creator.
That would at least allow users to back up their own creations which is a very important part of this.
In the future it would be good to expand the Inventory Backup to cover items made by other residents, either using current permissions or some new permission.
But as a start, and to get this feature as quickly as possible, Creator only backups would be awesome.
May 2nd, 2007 at 6:32 PM
Thank you for taking Project Letter so seriously. Second Life is a pioneering communications platform bringing the real world together and changing how we will one day utilize the internet in general. I commend Linden Labs for their efforts.
Unfortunately there have been some standing problems which the community is now asking you to resolve. We ask that you listen to our voice and address our concerns as we are your client base.
We who invest time and money into this platform because we believe in its potential feel that we have been largely ignored. We are asking not for quick fixes but a solid solution to these problems.
These problems clearly stem from the population explosion. Linden lab must come to terms with the impact this is having as a whole and find ways to regulate the endless growth Second Life is experiencing. We cannot deny LL its right to grow and expand and profit but Linden must find a way to limit or police it’s growth.
Perpetually free accounts are at the root of this issue. No other notable platform gives more than 14 days to a month trial time. Yes new citizens come, learn, explore and enjoy, but once you have decided to stay…upgrade… or leave.
As the article above mentioned Second Life requires ” manual rebalancing after periods of heavy Second Life growth.”.
I entered Second Life when the community was at 600k. I saw what Second Life was before the population explosion. Since then the grid has become a world of noobs living in noob slums without direction or sense. The Golden Age of Phil Rosenthal’s Utopia dream appears to have passed.
The population growth has homagenized Second Life. Where are the new Sverga’s, Arcana Nuevo’s, Ivory Tower of Prims, and Walden lakes?? Where are the road and railways, lakes, dams? Where is the beauty in all this?
Linden Labs has much to consider in the near future. It needs to decide what is more important: staggering population statistics to wow the investers, or maintaining a loyal client base who is content with the services provided. As the saying goes, “a bird in the hand is worth two in the bush”, Mr. Rosenthal.
I *IMPLORE* Linden Lab to resolve long term standing issues caused by overpoulation and not simply offer short term quick fixes to shush and pacify the community.
Thank you to all for reading this. I would like to end by saying that I truly believe there is resolve ahead and hope a new era will yet unfold in Phil Rosenthal’s vision and that all 6 million of us one day look back and can say “Wow, Linden really came thru for us after all”.
Cory, good luck tomarrow and thank you in advance for your time and considerations.
LeVey
May 2nd, 2007 at 6:32 PM
@27 - I concur. I have wanted to backup my own objects/scripts/textures for the longest time. At the moment, I manually have to do this piece by piece and have been slacking. It would be great if we could export/import an object as XML with base64 encoding for binary data and all it’s contents that we either created, or have copy/mod/xfer permissions on. Inventory management really needs a make over. It would be great to search/sort by creator, attachment point, or even get an idea how many prims are in an object before attempting to rez it.
May 2nd, 2007 at 6:34 PM
One Correction to Cory:
There is nothing wrong with “Find”. It was renamed “Search” several versions ago.
Seriously, maintaining consistent terminology In-world, in blogs in wikis, in help files and in other support documents is VERY important in providing customer service and support. Linden Lab (note: no “s” on the end) has been quite inconsistent in this regard.
Max
May 2nd, 2007 at 6:35 PM
Thanks Cory - we know this is a work in progress and we appreciate that sometimes things are going to bork as you guys keep working on it. I’d just like to echo the spirit of some of the posters here - a weekly (even bi-weekly!) ’state of the world’ briefing like this would take a lot of frustration away. Would that be possible? See you at the town hall
May 2nd, 2007 at 6:36 PM
Nice answers, thank you.
What is a real shame is that it took such a thing as project open letter to get a proper answer. To reiterate some things I’ve already said, there seems to have been a strategy aimed at disconnecting residents from Lindens; first live help went, then due to the online/offline options we could no longer see which Lindens were available, and really your phone system is a JOKE. I spent two and a half hours trying to get through to people and left a few “voice tickets” and have had absolutely no reply.
The main reason Project Open Letter came about was setting up an external website that shouted LISTEN TO US! was the only way to talk to any of the Linden staff, and if you’re going to offer any kind of customer support, that CANNOT continue. We NEED to have a line of communication open for issues that crop up that can’t wait indefinitely for an answer.
For now, it would go some way to solving the problems if the message of the day on login also mentioned two “on duty” Lindens for that time period - is it too much to ask for at least ONE direct line of communication to be left open?
May 2nd, 2007 at 6:40 PM
Hi Guys,
Cory i hope this isnt just another here the goodies we’ve got comign for you! But thanks for some concise info, thats all it took to keep us happy.
Whats really interested me though is the jump from HAVOC2 - to HAVOC4 there wsa no mention of this but quite amazing, what new prims limits will this give us on sim’s? Oringinally way back in a posting, Zero Linden i think it was said Havoc 2 could increase sim counts to 30,000 from 15,000? and ideas?
FOR THOSE OF YOU WHO DONT KNOW WHAT HAVOC IS.
Havoc is a physics engine that controls how prims interact with reality so to speak, eg particles, gravity, physical objects - wind etc…
HAVO 4 sample video - have a look at this to see how HAVOC 4 looks, quite amazing,
http://gamegreen.blogspot.com/2006/09/havok-40-demo.html
Thanks again Cory, so great info.
May 2nd, 2007 at 6:40 PM
I have a better Idea LLABS lets us be responible for our own Inventory ( so you dont have to be . since you can`t keep current levels od users inventory safely). Let us hold out own inventory on are our disk spaces. If you WILL NOT lets us back up are our inventory or even figure it out for it to happen( you will not even TRY to figure ot out with al the great schooled coders you have :/) I not bashing here just simple questens to answers the coders are suppose to hae the basic ABC`S about am i right? new virgin ears we are dealing with on sl, still does not pardon you from your responibities to game….Many of us work hard to set in bug reports and others issues you have over looked. But if the stream of questens and bug reports are not even being looked at or even thought ( Ingored ) or worse ( look at bugs reports that are totally useless reports ) about why should anyone.
To build a new world one need to learn
1.) What is the reason to improve things
2.) The usefulness of people and their abilities to deal with issues on the fly
3.) Hear the users and NOT ignore them
4.) Hire all levels of bug and grid monkeys not only whos ass you kiss to get the jobs
5.) Stop the all BOY`S/Girl`S Clubs when hiring in all level the game etc…
Good business is not SONY, EA, etc………What is LLABS is possiblty of building something that will be very special that will be with us for many many years……….”BUT IF” you stop the way you currently handle hiring coders and other grid type personal! WHo you know when hiring does not = the over all effectiveness of the outcome! BASIC BUSINESS 110 classes here ( did some staff in your ranks skip this we getting their masters and PH.D Degress……..:/
May 2nd, 2007 at 6:45 PM
@30 I was hatched with population was 30,000…………..
its was a small world and it was differnt
and now with all these members over 30,000 online at one time = the total popualtion then. tahts how far we have reached 
its great but we need to do alot more work before we be settled here cozy like :).
May 2nd, 2007 at 6:47 PM
Don’t forget to contribute to and visit the Town Hall forum thread set up by Jeska Linden at: http://forums.secondlife.com/forumdisplay.php?f=341.
May 2nd, 2007 at 6:56 PM
I think this is a good response. Communication def. needs to get better though- the blog is nice but it’s becoming more so ‘we are aware of a problem and are fixing it…’. We haven’t had a town hall in months. I’m not gonna blame that on conspiracy or anything, I think that when a company like LL grows very fast and a ton of issues pop up like have happend the past few months, its difficult to face the people when a fix may be right around the corner. The problem is, much of the time the fix has issues of its own so there is no ‘light at the end of the tunnel’, we need to know where we are NOW. Jira is nice but not a lot of people read jira, which leaves this blog and town halls.
May 2nd, 2007 at 6:56 PM
@37 I agree with you and believe our aurgument is generally the same. There is a lot of work that needs to be done to make this mechanism right again. Linden needs to police its growth and make good on what was promised.
I know many 2, 3 and a few 4 year olds. Your numbers are diminishing and even by todays standard I am an old man. It was because of citizens like you that I am so passionate about seeing Second Life redeemed.
Your passion for second life is commendable! I hope the efforts of you, I and many others like us can stear this thing back into better waters.
May 2nd, 2007 at 7:00 PM
@17 >>Have you ever seen the Beta grid at all? I was off in the Voice Beta last night and was very saddened to see less than 20 people there.
May 2nd, 2007 at 7:02 PM
Again, 80% of the people polled have indicated they do not want voice. That is 8 out of every 10 people polled that do not want voice. And that is the consistent poll percentage in polls and blogs, not just in SL, but throughout all third party forum polls and blogs.
So why are you surprised that you only see 20 people in the Voice Beta grid?
Pro-voice advocates refuse to heed to the writing on the wall. SL was doing great and flourishing with a text based communication system that was being easily supplemented with voice programs such as Teamspeak, Ventrilo, Skype etc. I don’t know what got into LL to want to suddenly jump head first into this voice integration thing, but lets hope the 20 people average on the Voice Beta is not an indication of things to come in SL.
May 2nd, 2007 at 7:03 PM
I don’t know what everyone is so happy about? All this basically says to a non-techy like myself is…Yes we know there are problems…and eventually we’ll get to them…
Look since the great leap to 1.14 My computer as been crashing on a regular basis. I was really thrilled to ear I was “in a minority” that just super!.
Frankly I don’t give a rat’s ass if I can find my flip flop faster in my inventory.
I WANT TO BE ABLE TO LOG ON WITHOUT PROBLEMS!!!
How is that so hard to understand? 1.13 was working fine…you guys at LL fucked it up…FIX IT!!!
I’ll worry about my watch rezzing up my ass another day.
All I’m saying is take a step back for god sakes and look at what you have and stablalize it….one we’re there…then start thinking new project.
All the new goodys don’t me diddly if you can’t log on and stay on.
May 2nd, 2007 at 7:04 PM
Hi Cory my two cents, inventory limits, I think that all residents have a
avatar key, is it not possible to have the invent local with a simple
rights mask copy/mod/transfer, only 3 coupled on the avatar key.
No limits needed, and if the user loss his inventory that has become an user prob instead of a linden prob! Proberly you need a server-side cache per account, but that can be limited.
A nice tool to keep your inventory clean on your local machine would be the final touch.
The server just have to check right-mask against avatar key.
Other solution maybe a way to get the invent back from local cache.
Regards,
Adrian
May 2nd, 2007 at 7:07 PM
Not only did Cory explain why we are having problems, he offered an apology. I signed the Open Letter and wish to say “apology accepted”!! Now let’s move forward
May 2nd, 2007 at 7:07 PM
@40 many of us that been around for years still believes second life is save-able. Many of use have changed “HATS” from what we started with when we first was hatched on sl. The strong make the weak quit. I am far quitting like those on my age and that are older that are left. Passion wittin game keeps us here. Oh we fight fight with each other bitch at one another, disagree with one another. But really we all mean well. We just express it in our own ways.Every little bit of dissagreementetc does help us understand one another when hashing out thought about how to improve things. If all we see is positive then what happens to the NEG remarks and comments? Thats why I feel some LLABS workers should “STOP all the The sky is clear and pretty” BUT REALLY its ” DARK GLOMMY and totally disorangized” The the truth and as you say the truth will set you free.
May 2nd, 2007 at 7:08 PM
Thanks for finally giving us some in-depth information-that is definitely more helpful than just a short “someone is working on something”!
And those complaining about limiting Inventory-please do read the blog entry again.
>>>We do not have any projects underway to limit inventory sizes, although it has been discussed.
May 2nd, 2007 at 7:08 PM
Hello :_)
I agree with all the above post’s and right now, since every thing I can think of that I wanted to say, has already been said. Thanks allot everybody *smiles*
Oh I almost forgot, Thanks Cory :_)
May 2nd, 2007 at 7:11 PM
@44 agrees!!!!!!!!!!! time to move on “IF” LLABS makes all the right move from now on.
Usagi
May 2nd, 2007 at 7:14 PM
Thanks for posting this! I didn’t understand everything written (some of the technical terminology is unfamiliar to me) but the response showed that you guys are listening to us.
Additionally, I really liked that you went into so much detail on some of the projects. For the first time in a while, I feel confident that Second Life is experiencing temporary teething troubles rather than permanent endemic problems.
Keep up the great work!
-Arden
May 2nd, 2007 at 7:28 PM
Changing data carriers to Level 3 is a good move.
May 2nd, 2007 at 7:34 PM
Thank you for addressing these problems, and most importantly, for communicating to us that you are in fact working on them. Most of us are very patient and understanding. It’s only when there are so many remarks on the problems and no communications from you that you are working on them that we are dismayed with the customer service aspect of SL, and to wonder if we should continue as paying members. Just having this post from you returns all confidence in SL and Linden Labs. Thanks again, and please, in the future, do keep us informed that the recurring problems are being worked on in a long run solution. Thanks!!!
May 2nd, 2007 at 7:43 PM
@48–To expect perfection is to invite only disappointment. There will always be mistakes, always be conflicts, problems, etc. I believe this is the first metaverse of it’s kind, so who knows WHAT the right moves are?? Right for you? Or right for me? Or them? I think if the Lindens will continue to say “you’re right, this isn’t working well, lets try to fix it” then the rewards will be worth the labors. (BTW I just crashed again–but I know this too shall pass).
May 2nd, 2007 at 7:57 PM
1. Hire Kamilion
2. Allow additional inventory back-up either locally or via a seperate service outside SL.
3. Read our email support requests and respond, not with just useless links (unless the answer is indeed in the links)
4. Stop allowing landbots to steal land…I am still waiting for my 21,811 Linden refund from a well-known landbot avatar (stole land from two italian ladies…other land owners got together and bought the land back…google a bit and you will see the stories) that stole my land during a transaction. Multiple emails for support continue to be replied to with automatic responses rather than real ones. Why would you allow one of your customers to steal money from another? Why would you not take this very very seriously and take immediate action? It is real money! I paid real money for those lindens on my credit card…and I had to give them to a landbot?? Why is this allowed…and acceptable to your company?
5. Stop bringing in new ideas like voice until you have the current system running smoothly.
6. Stop keeping all the money for the few of you that run the place and go ahead and hire more people, particularly for customer support and grid stability. Get serious about this thing you have created…too much energy is being wasted by you and us trying to get problems solved that should not even be issues….lost inventory??? Give me a break!! What is the point of the game if you pay for or create stuff and it gets lost and cannot be found?? Isn’t that pretty much the whole point of the game??? Basic needs folks….please…focus on the basic needs of your customers.
May 2nd, 2007 at 7:58 PM
First listen to what #19 Kamilion is saying. check out the web sites provided. This person knows what they are talking about. I think with the recomended upgrades most of your problems and ours would be solved. Yes, you’ll have to spend some more money but better performance means more paying customers. If my 1st SL experience had been as laggy as it gets now, I would have never joined. Thanks for listening.
May 2nd, 2007 at 8:00 PM
MySql and scripts?
Like Dallas I’m am an IT professional.
I recommend that you consider database software that supports live fail over for high availability and either replication or log shipping to synchronize multiple servers and transaction rollback for data integrity. I believe there are also some that support partitioning a database across multiple servers to improve performance.
Sometimes you get what you pay for, and if it’s free, that’s what you get, zip.
May 2nd, 2007 at 8:03 PM
O_O… Havok 4? skipping it all the way to 4?
That would be a big jump from Havok 1 instead of Havok 2
….O_o damn. Havok4 has a stronger support to swarm physic… I’m gonna have to imagine what would grid attack (not that I’m hoping to see) would look like.
May 2nd, 2007 at 8:07 PM
Cory.. as you write the above about inventory issues being resolved by fixing the bad script and that things are fine, my sim shows 100+ items in “pending downloads” after a fresh restart of the sim and is still unable to rez. Also, previously there was a blog post about financial transactions moving to a separate set of database servers and becoming more reliable, yet as recently as April 30th I’ve had failing transactions when paying.. and still no way to tell what it was that failed.
Why can we not see these “improvements” in world? As far as I can tell, the few times I login these days, I login ruthed and without the IMs and inventories offered while I was offline.. then 2 logins later they finally catch up. With the exception of users randomly crashing, our sim was better performing over a year ago than it is now… and at the time it had more content and more activity.. we’ve removed a lot since your systems are unable to keep up with the “create your world” vision that your web page promises is possible in Second Life and we now have even worse performance.
It would be interesting to see how many residents would support a full shutdown of the grid for 1 week while LL “really” fixes the problems instead of patch after patch. I for one, would have the patience and understanding since patching a live system is always overly difficult and adds unnecessary complication. Yes, the Linden dollar would suffer.. but lets be real.. when money transfers fail regularly with no accountability or record, it’s already suffering.
May 2nd, 2007 at 8:08 PM
Grid stability?
What about Viewer Stability????
Somehow I don’t think it’s a grid problem that causes our viewers to either crash or hang our computers.
May 2nd, 2007 at 8:17 PM
@ 52
“52 Ericka Laasonen Says:
May 2nd, 2007 at 7:43 PM PDT
@48–To expect perfection is to invite only disappointment. There will always be mistakes, always be conflicts, problems, etc. I believe this is the first metaverse of it’s kind, so who knows WHAT the right moves are?? Right for you? Or right for me? Or them? I think if the Lindens will continue to say “you’re right, this isn’t working well, lets try to fix it” then the rewards will be worth the labors. (BTW I just crashed again–but I know this too shall pass).”
You missed the point and are looking at the statement the wrong way………..
May 2nd, 2007 at 8:20 PM
I cannot attend the town hall meeting since I work during the time it is held. If I could be there and give input I would like to say that whoever designs the programing really needs to include CLEAR instructions and/or OBVIOUS Controls. Example: Something as simple as responding to this blog should be easy. I searched in vain for a way to post a response to yesterdays blog. Many parts of SL are like this. Its as if the engineers forget that not everyone is as far up the curve as they are and they assume “everyone” knows how to manipulate the site functions. Sorry doesnt work for me.
May 2nd, 2007 at 8:22 PM
Very nice, glad to see the letter got some attention.
Still no mention of dead vendors and servers though. :/
May 2nd, 2007 at 8:27 PM
Inventory Loss: And the bad news Cory is your “fix” has broken stuff yet again. A friend has now lost 7 out if 10 items that she was wearing before the update, including a pair of shoes, her ao and all her useful HUD tools. They are showing in inv, but on the one that is copy, showing as UUID 0000-0000 etc. Over 2k’s worth to replace plus the time to replace all the data when she could be working on her business. She’s not a very happy panda, since also for her the issues this server update was supposed to fix ie Group IMs that you borked, remain unfixed.
So another fine update and fix messed up. This is what we mean. Stop rushing headlong into things and make sure they are tested and the consequences thought out before you implement them.
Also, from your comments solely in response to the Open Letter, it appears there was far more fixed and updated than the issues this server update was supposed to address according to the blog. This is essential information that should be conveyed, not just selected bug fixes. That way people have an idea what you’ve messed with when yet something else borks up.
It might actually help with the JIRA and tracking these bugs down too if people know what you’ve fiddled with and broken. Even if the idea of beta software to report on what is looking like an Alpha release of software is somewhat bizarre…
May 2nd, 2007 at 8:28 PM
I used the Voice Beta as an example. First Look / Beta grids have always been ignored by most people on SL. They would much rather Female-Dog about SL being bugged than actually DO some testing for it.
May 2nd, 2007 at 8:33 PM
Perhaps you should discuss and work on your customer service and billing/credit processing area as well. Seems when most call they get voicemail after 20 mins and Never get a response by voicemail messages left or by emails to your support. Your support and customer service are severely lacking which I’m sure has been pointed out numerous times and it only seems to get getting worse not better.
May 2nd, 2007 at 8:45 PM
Thanks for finally saying something on stability that wasn’t blamed on the user’s computer or connection.
I’ve been around long enough and have a rather good memory to just say: I’ll believe it when I see it. Very few of the plans that were announced after a resident uprising have actually been talked about again, let alone made it to the grid.
May 2nd, 2007 at 9:19 PM
Thank You! : )
May 2nd, 2007 at 9:28 PM
Dolphpun: “First Look / Beta grids have always been ignored by most people on SL. They would much rather Female-Dog about SL being bugged than actually DO some testing for it.”
Dolphpun, feel free to speak for yourself, but pleae don’t presume to speak for others, ok?? I’ve used both the firstlook and beta, so may I have your permission to express views now?
And, there’s a reason its called “beta” - if “most of the people on SL” have used it, its not beta any longer - maybe you should go back and look up the definition of beta before doing so much of your own whining.
May 2nd, 2007 at 9:38 PM
“It might actually help with the JIRA and tracking these bugs down too if people know what you’ve fiddled with and broken. Even if the idea of beta software to report on what is looking like an Alpha release of software is somewhat bizarre”
Nothing but a rat maz that leads in a circle………helpful well time will tell but really doubt it will last.
May 2nd, 2007 at 9:54 PM
@ 35
HAVOC 4 looks like the way to go in the very near future. Everyone Check out the video!
http://gamegreen.blogspot.com/2006/09/havok-40-demo.html
May 2nd, 2007 at 10:19 PM
Is there anyone besides me that thinks maybe some of the people that complain the most are the ones CAUSING the troubles? The massive inventories and outfits with hundreds of prims and complex scripts and HUDs and so on and so on… I almost never have any problems at all except when the number of people logged in is over this week’s magic number. (what is it now? 36K?) Try deleting some inventory, detaching some stuff and wearing simpler clothing. If it has to download from the database, or it moves, glows or makes a noise, it’s lagging us all, right?
Now, yes, LL has to make improvements, but other than the sheer number of people logged in, what’s so different than from the mythical glory days? IMHO, it’s the complexity, individuality and detail some expect. It’s like people have rented a car and expect it to hold more passengers, go faster and look better with every day that passes by. People have unreasonable expectations.
The reality, in RL and SL is that there is only room for so much STUFF. At some point there’s a limit. I agree with whomever suggested ARCHIVED inventory items, ones that don’t download to your client unless you call for them. After a certain number of items, or voluntarily, have people archive, or store items. (like renting a storage space in RL) Should they pay to store ‘extra’ stuff? Perhaps.
Please take this in the spirit intended… I offer questions, not answers. But in the end, would it not help us all to simplify?
May 2nd, 2007 at 10:35 PM
Reply to #17:
if there was a “sandbox-weapons testing” and/or a combat sim on the beta grid is possible that lot of people would go there. A few thousands bullets flying, a dozen followers, a few nukes, a few particles generators, and an lot of other types of weapons is needed to really test how a sim behaves under heavy load.
Reply to #19:
scripts and/or physical objects not only can cause lag but can even crash the sim.
Reply to #29:
if u are a builder u can export/import your builds with the script from “Prim.Blender”. I also have something similar, but is kind of messy.
Reply to #31:
scripts are very easy to backup, u just have to copy and paste between SL viewer and an external text editor. I understand your pain, that is what I have to do with all my scripts.
Reply to #34:
is that a joke. Only people with PIOF have access to forums. How many users have that?
May 2nd, 2007 at 10:36 PM
MySQL? For something this large and intolerant to fault and corruption? You have to be kidding.
If you’re hell-bent on a free DBMS then look into PostgreSQL. In fact, Linux Journal has a great article about it in the May issue and compares the two in the June issue. MySQL is great for small-scale stuff - especially if you’re not too worried about data integrity - however, if you actually care about the data, opt for the DBMS that favors reliability over raw speed.
May 2nd, 2007 at 10:45 PM
17 Dolphpun Somme Says:
I feel you have no right to complain about poor testing if you are unwilling to even try the beta grids and first-look clients.
This is only My Humble Opinon.
64 Dallas Seaton Says:
Dolphpun, feel free to speak for yourself, but pleae don’t presume to speak for others, ok?? I’ve used both the firstlook and beta, so may I have your permission to express views now?
A) You never needed my permission.
B) I said that I feel it should be a specific way.
C) A the bold’s above indicate I am speaking for Me. Not you.
Beta testing comes after alpha testing. Versions of the software, known as beta versions, are released to a limited audience outside of the company. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number of future users.
64 Dallas Seaton Says:
definition of beta before doing so much of your own whining.
Oh, one thing I DID NOT DO is to make personal attacks on people for expressing their OPINION. I feel that generally people gripe without helping to DO anything themselves. However I would not be so rude as to call out specific people to prove my OPINION.
May 2nd, 2007 at 10:46 PM
LeVey Palou wrote:
“Perpetually free accounts are at the root of this issue. No other notable platform gives more than 14 days to a month trial time. Yes new citizens come, learn, explore and enjoy, but once you have decided to stay…upgrade… or leave.”
I strongly believe that limiting free accounts would really hurt SL, both in terms of the number of new paying members and in terms of SL businesses. There are MANY SL users who participate actively in the SL economy, purchasing Lindens and spending them throughout SL. Limit these accounts and you’ll be hurting thousands of businesses.
In addition, there are a number of users who simply rent land to start businesses. They’re also participating actively in the economy and might not bother if they were limited or forced to upgrade to an account level they don’t wish to upgrade to. Why shut them out? Do you really feel they’re not contributing in a useful fashion just because they’re not paying LL a monthly fee?
Furthermore, it’s been my experience that a lot of SL users who do upgrade do so several months after joining SL. Not days or even weeks, but *months*. In many instances it takes that long before someone is ready to purchase land, and if their free account were limited to just a few weeks or a month or two, many would likely leave and never bother to upgrade at all.
And finally, there are many private islands where land can be bought/leased from the owner and no premium account is required. A lot of private island owners have come to rely on this policy and would certainly lose a lot of business if their client base was removed.
Even requiring that an account be verified may have a detrimental effect (without a credit card, it can be next to impossible to provide billing information in many cases).
It is my belief that the majority of free accounts that don’t end up contributing to the SL world and economy in a useful fashion are abandoned before long anyway. If the users stay, they almost always contribute value to the world in one form or another.