I am preparing to merge SAK-11544 into the trunk – I invite folks to take a look at the code and give a shout out if they see any “gotchas”.

The code structure in this branch is a little strange because I wanted not to disturb broadly used APIs so that I could produce a patch version for post-2.5 and post-2.4 without spewing tiny API changes over a bunch of modules – to ease QA concerns for folks who want this in patch form.
I have cleaned up the APIs in question in SAK-12837. SAK-12837 is already done and committed in trunk. So when I merge SAK-11544 into the trunk it will be laid out a little differently and much more cleanly than the patch version in the branches below.
However the code that does all the work to page in SQL will be exactly the same as what is in these branches and the pattern will be the same.
It is a simple concept – do paging in SQL rather than in memory. Previously the mail tool read all the messages into memory and kept them in the session for *every user* – for sakai-dev this is 25K messages which would be 25- 50MB per user in session – ick.
The work also makes it so that a reindex of a site with a large mail archive does not crash the system.
This work makes *no* changes to data model – it just uses the current data model as efficiently as possible.
You will also notice that in the branches – there is still debugging – this will all be removed before my final patch versions for 2.5 and 2.4 are produced and will be removed before the trunk merge – they are there for you to see what is going on and to make some tests if you like.
This code has been tested by Tony Atkins (2.5) and Thomas Amsler (2.4) with good performance improvement – and no more servers out of memory / overloaded sessions. Stephen Marquad will help test the trunk code after merge.
So unless you have some strong concerns or see something that should be improved – I will start moving the two patch versions and the trunk version to final release – removing all debugging and then doing a final round of tests.
One final question to consider (perhaps in a different thread) is whether this should go into 2-5-x and then into 2.5.1? There is a slight functionality change in this code – when the MailTool detects that there are too many messages to efficiently sort by subject or from, it gives an error saying “too many messages to sort by subject”. So that is a functionality change. The functionality only goes away above an admin set limit of messages in the archive. And frankly, the UI that folks saw in these large archive sites was likely a traceback anyways :)
It has been suggested to put this in 2-5-x – I agree on the grounds that it keeps servers from crashing. We can set the property to a high number like 1000 messages so the functionality reduction is almost never seen – at the cost of chunky sessions – just like the current situation. But if folks feel that it should not go into the 2-5- branch – I am totally cool with that – less work for me :)
Feel free to pore over the code and the JIRA.
http://bugs.sakaiproject.org/jira/browse/SAK-11544
Instructions for patching 2.5 or trunk
rm -rf db message mailarchive search
svn co https://source.sakaiproject.org/svn/message/branches/SAK-11544/ message
svn co https://source.sakaiproject.org/svn/mailarchive/branches/SAK-11544/ mailarchive
svn co https://source.sakaiproject.org/svn/db/branches/SAK-11544/ db
svn co https://source.sakaiproject.org/svn/search/branches/SAK-11544/ search
/Chuck