The following is a transcript of the Technical Town Hall with Cory Linden, held on December 20, 2006 @ 2:30 PM PST.

Cory Linden: Good afternoon and thank you all for showing up. It has been far too long since I’ve done a town hall meeting so this should be fun. With Jeska’s help, I’ve pulled questions from the blog thread and I’d like to start by answering a bunch of those in a data-dump kind of way, then we can spend the bulk of the hour trying to hit as many additional questions as possible. Let’s see if we can’t set a new record for number of lines of chat text in one hour.

OK, here we go . . .

Lots of questions about scaling, free accounts, and data base performance. Most of our general scaling problems right now are keyed to our databases. As most of you know, we use a ton of MySQL databases running on Debian machines to handle lots of centralized services. Rather than attempt to build the one, impossibly large database – all hail the Central Database – or one, impossibly large central cluster – all hail the Cluster – we instead adopt a divide and conquer strategy based around data partitioning.

The good thing is we tend to associate UUIDs– 128-bit unique identifiers – with most things in Second Life, so partitioning is generally doable. What we have not done yet is the write a generalized partitioning layer above all of our data. eBay just
published (http://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdf) their architectural history and one important step they made was to build that abstraction layer. For right now, we have a finite number of additional partitioning steps to take until we have a lot of head room on most of the databases, but eBay’s approach is definitely interesting.

We have also been moving certain services off of MySQL – or building caching between the queries and MySQL. Presence, in particular Agent Presence, ie are you online and where are you on the grid, is a particularly tricky kind of query to partition, so there is now a spiffy Python service running on the SL grid called Backbone. Backbone is our service infrastructure of the future, since it relies on public protocols (HTTP, XML), is scalable, can handle lots of queries per
second, and can be changed without downing the grid (more on this in a bit). Presence queries for agents and simulators had been our high database nail, so we built Backbone around those problems.

In general, and you will hear this several times today, our engineering efforts are focused on moving away from proprietary solutions to existing web and net protocols that do the same thing. Within the next month, for example, we will be testing a new pipeline for texture downloads to the SL Viewer that just uses HTTP rather than the current messaging system. As you can probably guess, as l33t as the message system is, a fair amount of engineering has gone into making sure HTTP works over the net, so it is a little silly that we don’t leverage that. “Why didn’t we use it from the start?” you ask. Well, mostly because we made an error in estimating how many textures would be in the system. As a result, we optimized the messaging system around trying to save a few bytes here and there rather than realizing that over half the total data transmitted would just be compressed textures. This is similar to a class of errors related to underestimating what percentage of the total traffic would need to be reliable. Hence, we built a reliable protocol on top of UDP rather than having a TCP pipe available as well. Fortunately, we’re working on that now.

Regarding free users, they really aren’t too much of a technical load. There is a separate discussion related to their behavior and usage of SL– both of which I suspect are better than most suspect, but that isn’t really my area of focus – but from a scaling and technical standpoint they aren’t what worries me.

What does worry me? As you all know, Find queries are our new high nail. Beyond that, despite the many cries of “Please don’t make find better, it is already too good and way more than we need!” it is time to make search in Second Life really work. We are in the early technical and interface design period on search, but expect to be hearing more about it in Q1 2007 with a goal of rolling new search in Q2. Beyond that, we have to broadly scale SL’s architecture, but we’ll get to that in future questions.

Several questions about stability . . .
Stability questions actually break into a few different categories: server scaling, server stability, viewer scaling, and viewer stability. Server scaling, other than the centralized services issues I just covered, is primarily a function texture prioritization/downloading, script processing, physical simulation, and interest list calculation.

Per the previous discussion, we’re currently working on moving the texture downloads to HTTP, which will take a bunch of work out of the simulator. It will still be load on the machine, but it should be far less, especially in terms of disk access. Script processing I will get to when we talk about Mono. Physical simulation I will get to when we talk about Havok. Interest list calculation has been slowly improving over time. If the simulator gets too loaded, it doesn’t work very well,
can generate lag, and is more likely to lead to a bad experience, so you’ll see that we’re working along several fronts to make improvements there.

For the last couple of months, we had a crash in messaging between machines that was killing simulators in a particularly bad way. We’ve found the crash and deployed a fix, along with additional memory leak fixes, so simulator stability should be moving the right direction again. We still have more Havok 1 crashes than I like, but we’ll talk about that later.

Viewer scaling, which was also asked about in the SL 2.0 context, is an ongoing process. We don’t currently have the development resources to split off a team to start building a new client – although if anyone has a 10 to 20 person development team with experience in high-performance OpenGL who would like to talk, please email me! – so we’re doing the hardest possible thing, which is to rebuild the viewer as we go. Changes like moving textures to HTTP, multithreading, VBOs, and improving the oct-tree culling/occlusion code are all underway and will continue to roll out in the next few months. Especially in the brave new world of everyone having multi-CPU and multi-core machines, this has the potential to make a pretty big difference on absolute performance, but even more importantly, to reduce the stutter that is such a part of the existing viewer experience. It is always exciting using cutting edge graphics features, because since the SL residents aren’t hard-core game players, it is always an adventure to get drivers and configurations setup correctly.

That leads us into viewer stability, which continues to be a struggle as we push into more up-to-date OpenGL features. We have, as many of you know, started pushing more Beta and test viewer releases and are expending a fair amount of development horsepower to ensure that we can expand that program, but more on that in a bit.

Cory explains more technical details about Second Life
[14:36] Cory Linden: Again, we don’t see data suggesting that free accounts have a significant impact on any of these issues from a technical viewpoint. This leads to some great questions about testing, namely whether or not loading up an island or two in the test grid would help and
[14:36] Cory Linden: whether we can script the load of 15,000 avatars. In the first case, not really. Usually when we step on ourselves on a release, it is something related to full scale load, so even a few additional islands wouldn’t really help. As for scripting that full scale load, we
[14:36] Cory Linden: don’t have a good way to do that and none of the external QA folks we’ve talked to had good suggestions. These problems are what is driving our major rearchitecture effort, but more on that in a bit.
[14:37] Cory Linden: Avatar puppeteering is coming along nicely. Basically we’ve been doing a lot of UI testing, security testing, and bandwidth reduction. Speaking of bandwidth, a question about what we’re going to do to reduce bandwidth the client. Well, we’re not going to run on narrow
[14:37] Cory Linden: band any time soon, but we do eventually want to send larger chunks of simulator state downstream. This would have the advantage of allowing the viewer to be smarter about occlusion and culling, to make fewer state changes, and allow much higher compression on the
[14:37] Cory Linden: data. This isn’t even fully in the design stages, though, so this isn’t going to be worked on until H2 of 2007. HTML and Firefox . . . ah my two favorite topics of all time. We have an external contractor who has tons of experience working on it
[14:37] Cory Linden: (note that you can use the chat history to catch up if you fall behind)
[14:37] Cory Linden: right now. Basically we’ve been trying to make sure that we can get Flash working correctly because so many of the interesting parts of the Web are moving to Flash-based players/plugins/etc. Getting the control inputs and updates to work correctly is a bear but they do
[14:38] Cory Linden: seem to be making progress, which is very exciting. The order of operations will be to roll a full internal browser first, then supplement the parcel media types with URLs, and then move to full HTML on a prim. Note that HTML on a prim has several pieces, from
[14:38] Cory Linden: being able to interpret straight HTML in order to build text, do layout, etc, all the way to having a face of a prim point at a web page. In terms of timeline, the next major Firefox roll out will be in Q1 – ie, more functionality in the existing pages that use it plus a
[14:38] Cory Linden: floater that is a browser – followed by the parcel URL in Q2. HTML on a prim will be part of a larger rearchitecture of textures – we need to go to materials per face rather than texture per face – which several of the devs are itching to work on, but will realistically
[14:38] Cory Linden: not start until Q2. Now, the Amazon web services guys (and others) have already been using HTML to build text and then leveraged libSL to upload textures and
[14:38] Cory Linden: such, but we really want to make that easier. Speaking of libSL, there were some questions about why Linden Lab supports libSL. These are excellent questions. First of all, we can’t
[14:38] Cory Linden: stop someone from attempting to reverse engineer Second Life. We could drive them onto dark nets by aggressively shutting down public forums, club them with the DMCA by adding homeopathic quantities of encryption, and/or engage in an arms race by constantly changing our
[14:38] Cory Linden: protocols. Does anyone really want us to spend time doing those things? You’ll notice again and again that when we talk about features, bug fixes, or scaling issues that we have with SL that we are limited by development resources. Developer time is precious and trying to
[14:39] Cory Linden: (note that all of this will go out to the blog, too, but by doing a data dump we can hopefully avoid duplicate questions)
[14:39] Cory Linden: (if you are missing on the repeater, apologies and we’ll make sure it goes to the blog)
[14:39] Cory Linden: chase the libSL folks around would be an infinite time sink. Nor does that take into account the positives. The libSL folks have reported literally hundreds of exploits to us rather take advantage
[14:39] Cory Linden: of them. Yes, there are members of their community who have done dumb things but what community hasn’t? I would much rather have that army of talented developers spending most of their time trying to make SL better than either driving them away or being in constant
[14:39] Cory Linden: conflict with them. If they break the terms of service we treat them like any other resident, but just because they tinker does not make them criminals.
[14:40] Cory Linden: As we’ve talked about, the long term goals for Second Life are to make it a more open platform. Part of that process is learning how projects like libSL can be beneficial to all of Second Life. We should be thrilled that we’ve built an interesting enough set of
[14:40] Cory Linden: technologies and communities that people want to tinker and explore.
[14:40] Cory Linden: In the long run, this is why we’ve talked about wanting to be able to Open Source eventually. My hope is that in 2007 we’ll be able to get there.
[14:40] Cory Linden: As long as we’re on libSL, one of our devs has used it pretty extensively in testing Mono. We’ve had several full up tests of Mono
[14:40] Cory Linden: running on grids and things are looking very good. Because the Mono project is still dealing with some security issues and we’re finding
[14:40] Cory Linden: memory leaks both in our changes to Mono and their code, we’re moving slowly and carefully. Novell/Ximian have been really helpful, so
[14:40] Cory Linden: we’re making decent progress. There are some technical changes we still need to make in particular, we’ll need to compile Mono on the
[14:40] Cory Linden: server side which requires a distributed compilation service to be running on the grid (yay, backbone!) but I expect that we will begin
[14:41] Cory Linden: testing Mono on the main grid in Q1/Q2 2007. The process there will be to have places on the grid where you can bring scripts and
[14:41] Cory Linden: recompile them into Mono for testing. That will let you report broken scripts to us. Since Mono tends to execute LSL about 600 times
[14:41] Cory Linden: faster, I expect that there will be some interesting borkage around carefully timed scripts.
[14:41] Cory Linden: Babbage has talked about the implications of Mono extensively, but it’s important to remember that the sequence will be:
[14:41] Cory Linden: 1) Start allowing compilation of LSL to Mono/CLI. Test existing scripts like crazy. (Q1/Q2)
[14:41] Cory Linden: 2) Think about ways to include other languages (Q more than 2)
[14:41] Cory Linden: Regarding Havok, we are once again considering how to use a version of Havok with a version number greater than 1. Stand by for more info Q1/Q2.
[14:41] Cory Linden: There were several questions around search. We’re starting a pretty major design phase about search to look at how to make it not suck
[14:41] Cory Linden: (as much.) We can talk about that more if you want to, but basically I know that we can make both the new user and regular user
[14:42] Cory Linden: experiences way better than they are today. Design work starting there in Q1.
[14:42] Cory Linden: Definitely a good point that we’ve been a bit heads down in dev of late, mostly attempting to get ahead of scaling issues. I will attempt
[14:42] Cory Linden: to do more town halls next year and begin blogging again to keep people aware of our timeline and goals. Generally, next year is about
[14:42] Cory Linden: (apologies for the borked repeater, we’ll post all of this to the blog in a moment)
[14:42] Cory Linden: making the existing stuff work better, scaling more, and rolling out features that I’ve talked about today. A few quick answers:
[14:42] Cory Linden: I like the idea of differential upload costs for textures, I’m curious what other folks think about that. The thought is that it would
[14:42] Cory Linden: cause people to be more careful about picking appropriate sized textures.
[14:42] Cory Linden: We only have Quicktime for video playback because when we put it in it was easy to add to both the Windows and Mac versions.
[14:43] Cory Linden: Yes, adding either some sort of curved surface or different editing tools would open up the design space and reduce the number of
[14:43] Cory Linden: We went with llSetPrimitiveParams because the features it was impacting were changing and we didn’t want to keep deprecating functions.
[14:43] Cory Linden: Yes, adding either some sort of curved surface or different editing tools would open up the design space and reduce the number of
[14:43] Cory Linden: “squarish” objects.
[14:43] Cory Linden: I love the idea of a technical details website. Clearly, much of our outbound messaging about stuff is woefully out of date. Of course fixing that is a lot of work . . .
[14:43] Cory Linden: Speaking of hiring . . . we are reading all the resumes that come in. We are still looking to hire, but understand that if your skill set
[14:43] Cory Linden: doesn’t match what we are looking for that we will likely not ever get back to you. We are trying to improve the feedback in that system,
[14:43] Cory Linden: but we’re working on it. Also, to reinforce what Philip said, if you and a friend are somewhere other than the Bay Area, we still should
[14:43] Cory Linden: talk. We are already a distributed company and I see no reason why we can’t become more of one.
[14:43] Cory Linden: Finally, Gwyneth had a very long set of questions that I can’t possible answer all of. I’ve hit a few of the points already, but broadly
[14:43] Cory Linden: I will say that we are trying to update our messaging system to more properly leverage existing standards rather than our hand-rolled
[14:43] Cory Linden: one. A big part of the engineering team has been building the pieces of what we call rapid deploy, which will ultimately let us roll
[14:43] Cory Linden: pieces of the grid at a time, not force updates, and let us deploy test code to the main grid where people will actually hit it. Doing
[14:44] Cory Linden: this is, as you might imagine, a tremendous amount of work, because we have to start by taking message template out behind the shed and
[14:44] Cory Linden: shooting it. Then, we need simulators and viewers to be able to discover what features they have, be able to fall back gracefully, etc etc.
[14:44] Cory Linden: Also, I would ask folks to remember that when we undertake major changes, it is a little like trying to rebuild the engine of a car
[14:44] Cory Linden: While driving it
[14:44] Cory Linden: And changing to a hydrogen economy
[14:44] Cory Linden: Without hitting anyone, losing control, or breaking the car. If I had 50 spare developers, I would be starting parallel projects, but hiring is tough right now.
[14:44] Cory Linden: ok, that complete’s the data dump, apologies for broken repeaterage
[14:44] Cory Linden: hold on a sec while I email it around to get onto the blog
[14:45] Jeska Linden: Ok, I’ve got a slew of ‘em for you.
[14:46] Cory Linden: ok, the billion people IM-ing me are not helping the process
[14:47] Cory Linden: Jeska, do you want to just toss out questions and I’ll do my best to answer them?
[14:47] Jeska Linden: Yup, first one

Question & Answers
[14:47] Jeska Linden: Teravus Ousley: The Super-Monsterous prim .. that are larger then 10×10x10 - Some the size of a sim and larger. What is the current action if a Linden comes across one of these prim (that isn’t affecting someone else’s land. Do you leave it, delete it, etc..
[14:47] Jeska Linden: Don’t forget everyone - join the group “Linden TOwn Hall Questions” to ask your question.
[14:48] Cory Linden: that’s not really a technical question, so I think the policy is to return them if a nuisance
[14:48] Cory Linden: next

[14:48] Jeska Linden: Howie Lament: Question: Have you thought about separating the chat interfaces and other widgets from the 3d view, and use OS widgets for those parts? It would help keep the UI responsive and easy to use in high load situations such as these.
[14:49] Cory Linden: the ui question is an interesting one, although a little tricky to integrate with an opengl app cross platform. We almost went that way 5 years ago. The good thing is that the XML updates to the UI make the possible someday if it seemed like the right direction.
[14:49] Cory Linden: next

[14:50] Jeska Linden: Travis Lambert: Q: Sensors are incredibly inefficient due to detected* bloat. As a result, we have to use many sensors to cover a sim, or to pull back basic Key info for more than 16 avatars, increasing simulator load. Is any work being done to give us some more optimized versions of highly used (and laggy) LSL functions, that could have the effect of reducing sim load? A llGetAllAvatarsInSim() type function combined with the existing llOverMyLand() could remove the need for a lot of sensors out there. Your thoughts, Cory?
[14:50] Cory Linden: ah, question was about creating new lsl functions that are optimized for specific uses.
[14:50] Cory Linden: seems like a good idea, perhaps start a page on the lsl wiki that sorts the ones that would be most useful. that would be helpful.
[14:51] Cory Linden: next

[14:51] Jeska Linden shouts: Lewis Nerd: Question about linksets - why the 31m limitation on end-to-end dimensions on linked regular prims? There are many legitimate uses for larger linksets. If it’s to do with physics… why is this limit still there when physics is not enabled? Is it something that a future Havok implementation will eliminate?
[14:52] Cory Linden: QUestion about size of link set limits. Why are they there? They are there for historical reasons of interaction with Havok and deep think. Yes, we would like to fix that in the future because it is really a problem. Plus, we need to think more broadly about how to organize prims beyond linking.
[14:52] Cory Linden: That is all farther out than immediate scaling and other things I covered in the data dump.

[14:52] Jeska Linden shouts: Lowell Cremorne: What infrastructure expansion process do you have in place to prevent a worsening of stability issues, and will the infrastructure actually grow enough to reduce the issues?
[14:52] Cory Linden: (for those IM-ing me, I am not answering IM right now)
[14:53] Cory Linden: Question about scaling and infrastructure. Will SL scale? I tried to cover that a bit in the data dump. We have a good plan to continue to divide and conquer the database by partioning data
[14:53] Cory Linden: so it’s a matter of keeping ahead of growth. We also got caught because extra bandwidth took longer to provision than we had expected, but now we know that one now

[14:54] Jeska Linden shouts: WindRider Taiyang: In the “Edit Appearance” mode, I am unable to orbit my av to check my progress. Is this a system wide problem? If so, are we looking at a correction soon?
[14:55] Cory Linden: Question about a specific bug in Edit Appearance: I don’t know if the orbit problem is general, but if it is please submit a bug report.

[14:55] Cory Linden: oh, a follow up on megaprims: we are going to allow them on private estates as well.

[14:56] Jeska Linden shouts: Partyhats Loll: What is LL doing about the griefer alt problem?
[14:57] Cory Linden: Question about what LL is doing about griefer alts… well, we are continuing to roll improvements to the group, land, and estate tools plus we are continue to build in better system-wide responses to large scale griefing. Identity is clearly not a solved problem on the web, so community and local control seems the only scalable solution.

[14:57] Jeska Linden shouts: Sweet Valentine: I would like to know when my missing inventory items will be returned?I filed many bug reports and i have a HUGE folder of missing from database that are older still waiting no one has contacted me about this ever I have many items i tryed to rez and they have just disapeared no one ever contacts after we bug report and they are never returned..hese items cost me ALOT of money RL money what is going to be done about this??
[14:58] Cory Linden: Question about lost inventory: we do our best to follow up on bug reports. Sometimes those follow ups will take longer than we like and for that I am very sorry. A big part of our internal scaling discussions are around how to scale those aspects of the business as well.

[14:58] Jeska Linden shouts: Nobody Fugazi: Regarding the internal browser: Will there be better hooks with LSL using the internal browser or will it be the same?
[14:59] Cory Linden: question about whether lsl will be able to hook into the embedded browser? great question. at first, likely not, but it seems like a real opportunity to make sl better integrate with the web (like someone asked about myspace) so we should think about that.
[15:00] Cory Linden: but it has someinteresting security implications if you can throw naked markup at a browser and/or trigger events on it

[15:00] Jeska Linden shouts: Zonax Delorean: When will we get credits/refunds for the so much instability and downtime? I pay over 400 USD/year, not on a free account!
[15:01] Cory Linden: A question about refunds and credits, which are spelled out in the terms of service.
[15:01] Jeska Linden shouts: Zi Ree: What is the *technical* reason for an avatar’s shape not loading (invisible), and why do you suggest people to update theis graphics drivers when that happens?
[15:02] Cory Linden: question about why avatars don’t appear… the avatars often exercise parts of the opengl driver that are cranky. Updating drivers for graphical glitches is generally a good starting point.
[15:03] Cory Linden: However, there are sometimes other reasons, such as slow down in the central db, asset server, or the sim you are on.

[15:03] Jeska Linden shouts: Tao Takashi: Would it be possible someday to reference textures just by URLs on external servers? Getting data into SL is a bit of work right now but would be very interesting for many projects (alternatively a webservice to upload textures)
[15:03] Cory Linden: question about referencing textures by url rather than adding to system… yes, that is something that we would like to do in the future, although we want to avoid a bunch of 404s everywhere
[15:04] Cory Linden: plus, we still need to think about how to properly prioritize textures pulled from the web since just going out and doing http gets to all of them would be, well, slow
[15:04] Cory Linden: really slow

[15:04] Jeska Linden shouts: Franko Corleone: Is there currently an issue in some areas relating to the inability to “edit” a notecard within an object (irregardless whether the object is lockked or not). as this issue has been present and reported in Lippert for 4 days now?
[15:05] Cory Linden: question about editing notecards in lippert… haven’t heartd of that one. Has anyone tried moving the object to another sim? does it work then?
[15:05] Cory Linden: ok, let me ask around and see what’s up with that
[15:06] Cory Linden: oh, so it doesn’t work no matter what
[15:06] Cory Linden: hrmph . . . ok, will poke people and see what I learn
[15:06] Cory Linden: can someone give me a copy of the borked object?

[15:06] Jeska Linden shouts: (i think he’s being silly… but) Kage Seraph: Cory, on a technical level, what is the mix of bananas to vegetable matter that the gridmonkeys are fed?
[15:06] Cory Linden: question about care and feeding of gridmonkeys…
[15:06] Cory Linden: depends on the grid monkey
[15:07] Cory Linden: some exist almost purely on beef jerkey while others require 15 year old single malt
[15:07] Cory Linden: no two grid monkeys are exactly the same and care is required in not making a mistake as that can be catastrophic

[15:08] Jeska Linden shouts: Orochimaru Eusebio: Hello I’m representing a group of Creators within SL who all relied on the WASD camera control functions, I would like to know why you removed it and if you plan to bring it back as Im sure youre aware alot of builders used this.
[15:09] Cory Linden: question about the wasd camera controls…they were removed in favor of the arrow keys in order to make it easier to get to the chat bar
[15:09] Cory Linden: alt+wasd works
[15:09] Cory Linden: perhaps that needs to be an option…
[15:09] Cory Linden: no, it doens’t work
[15:09] Cory Linden: sorry, a linden who will remain nameless just misled me
[15:09] Cory Linden: they will be punished

[15:10] Jeska Linden shouts: Chalky White: can we have big textures back please if we pay for the nose - I badly need 8192×4096. harmless surely and I have a real use.
[15:10] Cory Linden: great question about HUGE textures if you pay . . . hmmm, maybe, but I think that we need to solved that within the context of an overall look into upload charges.

[15:11] Jeska Linden shouts: Mister Arrow: Any word on seperating the Mainland and Estate land listings in the search. Just a filter would be appropriate as they are certainly more different than PG/MATURE and on par with Auction v First Land vs For Sale?
[15:11] Cory Linden: great suggestion to separate mainland and island results in search. Not sure if the existing COMPLETELY TOTALLY ROCKING search code will support that, but I like the suggestion.

[15:12] Jeska Linden shouts: Kingsbury Yeats: When will the open source client be released?
[15:12] Cory Linden: question about when we might consider an open source client … the hope is that we are able to do that in 2007

[15:13] Jeska Linden shouts: Jessica Elytis: As technology increases (not always as fast as desired), when, or how, is LL planning on matching that with management and personell as LL moves from a smaller company into a larger corperation? And what technical solutions are in the works for communticating with the Residents properly? Teh problem highlighted by the broken Repeater issue here today.
[15:13] Cory Linden: a question about ll’s scaling (as opposed to SL’s)… yes, one of the hardest questions is how Linden scales while still being a great place to work.
[15:14] Cory Linden: We are trying to be pretty innovative in how we organize development and are fortunate to have some great folks stepping up as we grow.
[15:15] Cory Linden: But as I mentioned, hiring is hard and we’re looking as hard as we can. Plus, if you have a couple of folks who are brilliant but not in the bay area, please drop us a line
but not in the bay area, please drop us a line
[15:15] Cory Linden: we are a very distributed company already and want to become more so.

[15:15] Jeska Linden shouts: Chalky White: Do you have a technical fix imminent for the camping/clubs making their neighbors land permnanently unusable?
[15:15] Cory Linden: a question about camping chairs making land unstable… we are looking to change dwell to eliminate the benefits of camping chairs.

[15:16] Jeska Linden shouts: Graciella Princess: What exactly is being done to improve stability and reduce lag and packet loss?
[15:16] Cory Linden: question about stability/lag . . . I went into a fair amount of detail in the data dump, so please start there. Broadly, we are partitioning the dbs,
[15:16] Cory Linden: and looking to change how we do deployments so that we can roll test code more effectively

[15:17] Cory Linden: oh and no I don’t have a Linden bear

[15:17] Jeska Linden shouts: Nobody Fugazi: Will application/xml be supported by llHttpRequest/httpresponse any time in the future
[15:18] Cory Linden: question about http request supporting application/xml . . . there are host of changes in how we fire off http requests that will be rolling in the q1/q2 as we continue to improve how we call out to the web.
[15:18] Cory Linden: comments in the lsl wiki about things that you need and why you need them would help that order of operations.

[15:18] Jeska Linden shouts: Talarus Luan: When will sim crashes be fixed? People pay a lot of money to be constantly crashing in their own island sims. Here’s an example of what I am talking about: http://www.iconia.org/sl/crashquery.php
[15:19] Cory Linden: question about when simulator crashes will be fixed … we had a serious bug through october and november that was crashing sims in the messaging code. That bug is now fixed, so sim crash rates should drop. As we find them we try to kill them.
[15:19] Cory Linden: We also are working on making sure that the crash behavior is better so that content isn’t lost

[15:20] Jeska Linden shouts: Geeky Wunderle: Is there anything we as residents can do to help during the time scaling issues are being worked on?
[15:20] Cory Linden: question of what residents can do to help with scaling and testing… when the beta and test grids are up, please use them! That testing is incredibly important to us and help a lot!

[15:20] Jeska Linden: Icy Eber: Why can’t people make custom Prims?
[15:22] Cory Linden: why can’t people make custom prims? hmmm…
[15:22] Cory Linden: we actually have a lot of thoughts on how to make the creation tools better
[15:22] Cory Linden: custom prims is an interesting idea, although the security and performance issues are tricky

[15:22] Jeska Linden shouts: Davina Glitter: What is being done now and will be done in the future to meet the high demand for SL?
[15:23] Cory Linden: question about general scaling…. I’ve already answered this.

[15:23] Jeska Linden shouts: Craig Cleghorn: Have the lindens considered beowulf(cluster server) or similar technologies, such as distributed processing to increase bandwidth issues, or possibly decrease lag issues such as
[15:24] Cory Linden: question about using beowulf clusters to solve our scaling issues… we are already
[15:24] Cory Linden: spactially subdivide the world and partition the dbs, so we’re effectively dividing and conquering already

[15:24] Jeska Linden shouts: Howie Lament: Is opening a parallel universe SL grid totally out of the question? If making SL scale properly is a lenghty process, how about opening new datacenters in asia and europe and let them be separate grids, then connect everything together when the infrastructure is ready?
[15:25] Cory Linden: we are already starting to open additional colos and eventually that will include more of the back end systems. I am not contemplating parallel grids currently.

[15:25] Jeska Linden: Eddy Stryker: Has the DTD for LLSD changed at all since it was originally published in July 2006?
[15:26] Cory Linden: question about whether the llsd dtd has changed . . . asking on backchannel will answer when I know, next please

[15:27] Jeska Linden shouts: Zonax Delorean: would closing all new registrations temporarily help with the increasing load?
[15:27] Cory Linden: no, closing registrations wouldn’t really help with the current load. Next, please.

[15:28] Jeska Linden shouts: Heather Goodliffe: QUESTION: I’ve been wondering for a while why LL doesn’t rollback updates when they turn out to have a problem (as most companies do?), since rushing out more updates can often lead to more problems?
[15:29] Cory Linden: question about why we don’t rollback on bad updates… we really really really REALLY wish that was possible but with the current archoitecture it isn’t. However, rapid deploy is one step towards that.

[15:29] Jeska Linden shouts: Clody Castro: I need a tool to backup my work om my personal compuer in a standardised format. Would you provide us such tool /menu option in the future?
[15:29] Cory Linden: question about backing up data… question in return is whether everyone is comfrotable with backup tools that can be used to duplicate data…
out to have a problem (as most companies do?), since rushing out more updates can often lead to more problems?
[15:29] Cory Linden: question about why we don’t rollback on bad updates… we really really really REALLY wish that was possible but with the current archoitecture it isn’t. However, rapid deploy is one step towards that.

[15:30] Jeska Linden shouts: Caesar Ah: ‘There’ sucks, but they do have voice chat, when will you have the same?
[15:30] Cory Linden: question about when we will have voice chat… a lot of folks use it already, but we could clearly do more to integrate it

[15:31] Jeska Linden shouts: Heather Goodliffe: Might it be a good idea to implement user quotas on resources (script time, bandwidth?, emails, IMs) based on account type, where people are throtled once they hit a soft limit and severly throttled once they hit the hard limit? Couldn’t this eliminate a lot of SL’s current problems with lag?
[15:31] Cory Linden: questiona bout whether user quotas would be a good thing… yes

[15:31] Jeska Linden shouts: Tao Takashi: Can we get more groups or in general better tools for collaborating on projects (svn for scripts would be nice?
[15:32] Cory Linden: question about better group tools… community team is working on designing those as we speak

[15:32] Jeska Linden shouts: Taft Worsley: Why is the inventory stored on a datbase server and not the local machine ?? would seem to me it would help with DB TPM and would prvent the lindens from being responsible for the inventory.
[15:33] Cory Linden: qquestion about why inventory is on the back end… local inventory opens up a host of issues but again, if one is comfortable with copying, then local inventory makes more sense

[15:33] Cory Linden: ok, thank you all very much for the great questions
[15:34] Cory Linden: will attempt to not go 6 months before the next one
[15:34] Jeska Linden: Thanks for coming to the Town hall everyone, sorry about the repeater issues.
[15:34] Cory Linden: They were great questions and we’ll do another one of these soon.
[15:34] Cory Linden: Thanks, Jeska