Dr. Chuck's Web Log

Compiling the kernel

Maybe you want to be the first on your block to make a patch for the new kernel. So you need to check out, modify, and compile the new kernel. You may even want to test your changes in Tomcat :)

When Ian wakes up, he might tell me that I am completely wrong - but this worked:

cd ~/dev/sakai-trunk

svn co https://source.sakaiproject.org/svn/kernel/trunk kernel

I check this out into the same directory as sakai - not inside of the sakai directory - so ls looks like this

charles-severances-macbook-air:sakai-trunk csev$ pwd
/Users/csev/dev/sakai-trunk
charles-severances-macbook-air:sakai-trunk csev$ ls -ld kernel sakai
drwxr-xr-x 23 csev staff 782 Jul 30 14:39 kernel
drwxr-xr-x 73 csev staff 2482 Jul 30 15:21 sakai

Then I go into the kernel directory and type this:

mvn -Dmaven.test.skip=true -Dmaven.tomcat.home=/Users/csev/dev/sakai-trunk/apache-tomcat-5.5.23/ clean install sakai:deploy

This is the same thing I type to compile sakai.

It compiles kernel - puts it into my maven repo and then puts it right into my Tomcat!

Voila!

My K1 Experience

I am back up with K1 - it was not so hard - My instructions are a little different than Ian's because I use vi and no debugger. A key here is that you don't have to switch to how Ian does his tomcat stuff - just for K1 - I did my normal process (different than Ian's) and it all still came up fine - so whatever your process is - it should work as well.

I include my steps

Whacked the repo (per Ian)

rm -rf ~/.m2/repository/org/sakaiproject

Whacked my tomcat

rm -rf ~/dev/sakai-trunk/apache-tomcat-5.5.23/

Unzipped a new fresh tomcat 5.5.23

cd ~/dev/sakai-trunk/

curl -O http://source.sakaiproject.org/mirrors/ibiblio/maven/tomcat/zips/apache-tomcat-5.5.23.zip

unzip -q apache-tomcat-5.5.23.zip

chmod +x apache-tomcat-5.5.23/bin/*.sh

Clean check out

svn co https://source.sakaiproject.org/svn/sakai/trunk/ sakai

Full compile attempt - Failed with the following message

[INFO] Failed to resolve artifact.

GroupId: org.sakaiproject
ArtifactId: master
Version: SNAPSHOT

Reason: Unable to download the artifact from any repository

org.sakaiproject:master:pom:SNAPSHOT

from the specified remote repositories:
central (http://repo1.maven.org/maven2)

Do the master thing - as per Ian:

cd sakai/master/
mvn install

-- BUILD Successful

Full compile:

mvn -Dmaven.test.skip=true -Dmaven.tomcat.home=/Users/csev/dev/sakai-trunk/apache-tomcat-5.5.23/ clean install sakai:deploy

Started tomcat - navigate to http://localhost:8080/

Viola - Sakai is back with K1

Now - I am wondering where site code went... Hmmm. I was busily working on SAK-13482 and was ready to check in - when I realized the world had just changed out from under me - that will wait until Ina gets back.

NICE WORK IAN!

Kernel 1 - In the house

Our trunk is now reorganized to define a kernel - this will be a few days of angst - the contrib folks will need to react - Ian has provided a nice XSLT to get the pom.xml files fixed with minimal pain.

This will also make the 2-5-x branch management a little more dicey for 2-5-3, etc.

Progress is needed and this is a great time to take this plunge - when we have 2.5.2 in hand - and the 2-5-x branch is in pretty good shape - and we have some time before the 2.6 code freeze.

Folks not working on the deep bits of Sakai - might want to avoid doing an update for a few days - if you are working in the new Kernel area - you pretty much need to update and help figure out the new way.

Here is Ian's note.

Again - thanks for the fine work, Ian!

K1 is now integrated in core as of r49761

There has been 1 casualty akaict, Message Forums wont start due to a
problem in PermissionLevelManagerImpl with the default permissions. I
have put a temporary fix in r49761 to allow Sakai to start (big
traceback in the startup). Sorry to the Message Forums team.

Since this is a major change you will need to clean out and rebuild.

svn update
mvn clean
# Its a good idea to clean out the local repo to ensure that really
do get fresh copies of dependencies.

rm -rf ~/.m2/repository/org/sakaiproject

#Gradebook tests are currently broken (have been for the past few
days), removing them will enable you to get a Full build with full
tests (such that they are)

rm -rf gradebook/app/standalone-app/src/test/

# Some version of maven can resolve ../master/pom.xml in a single
build, so build and install the master first

cd master
mvn install

# Do the main build, make 100% certain that your network connection
is *not* going to return a HTTP proxy authentication page, otherwise
you *will* get HTML instead of Jars at this point

cd ..
mvn install

# Make a cup of tea, eat a pizza, watch a 30 minute movie (as long as
it doesn't involve bandwidth)

# You *will* see kernel SNAPSHOT artifacts being downloaded from the
Sakai maven repo. You will also see some other sakai artifacts,
pulled from various repos including hanging M2 artifacts cause by
RSF. These are Ok (you zapped your org.sakaiproject. local repo)

# Create yourself a *clean* tomcat deployment directory.

mvn sakai:deploy -Dmaven.tomcat.home=/Users/ieb/Caret/sakai22/
tomcatkernel/

# This should deploy a Kernel assembly (downloaded from the maven
repo) as well as the whole of core.

/Users/ieb/Caret/sakai22/tomcatkernel/bin/catalina.sh jpda run

# Should startup, go to http://localhost:8080/portal ( and after
that you should see absolutely no difference :) .... that was worth
it, wasnt it!)

# In the logs, you will see the sakai-kernel-component starting up.

If you do a svn status you will see a bunch of projects no longer in
externals

rm -rf entity component event db util cluster jcr

will clean them up

I have tested the build multiple times now, and will be offline for
the next few hours, but will pick up again this evening to try and
help or fix what I have broken.

Ian

Sakai K1 Kernel Commit Policies

The Sakai K1 Kernel will be in our svn pretty soon! This is a great milestone for Sakai - it moves us toward a much more manageable source tree given the kinds of resources we have available in the community.

For me this is a welcome move back toward a "do-ocracy" - at least for this very technical bit of the code. Do-ocracy and dem-ocracy have their places - and in particular for Sakai - with a large,complex user interface that end-users touch daily - it is important to not be a pure do-ocracy. But for the deepest bits of our code - the kernel - pretty much Apache Rules need to apply here with no grey area.

Ian recently posted the following vote to the dev list - lots of folks want comfortable grey areas like we have had in the past out of necessity. As folk might expect from me - I prefer grey only when there is a reason for grey.

Here is Ian's question:

Going forwards I would like to adopt a more Apache like policy to kernel modifications in K1... namely

1. non kernel committers have to submit patches to Jira for review

2. commits < 10 lines that dont change functionality can be committed by kernel comitters

3. all commits > 10 line that change functionality have to be submitted as patches to Jira for review, review is light weight and commit is relatively fast afterwards.

If some one wants a branch, thats fine, but the above practice still stands, they need to create a patch and submit that patch to jira.

Please vote.
Ian

Here is my answer - pretty much grey-less as you might expect :)

+1

I would say that it depends on who the kernel commit group is and how much change is needed in kernel and how responsive the kernel commit group is to patches.

Historically, patches have a pretty high probability of being ignored - but also historically we have very shallow coverage of commtted commiters with time to spare on much of the source tree.  Many chunks of our code base do not really even have a "lead committer" - there are folks who work on stuff when their is something like a bug, etc.

This is not bad - it is just the nature of a million lines of code and <10 people who work >10 hours per week on the "problems of the commons".  Lots of people do lots of work in lots of places - here I am only speaking of those people who work on the problems that are not their local priority.  We are spread thin - too thin.

For me the point of the kernel is to change this pattern - at least in the kernel itself.  Hopefully we will have 2-3 lead/solid kernel committers who can be counted on to "tend" the kernel giving, say 30 hours per week total available maintenance time on the kernel.  It should not take 30 hours *every week* - but we need folks with long -term spare resources available on short notice to work on the kernel when things arise.

We will have a Kernel classification in JIRA - so there is one "input queue" for this kind of issue / problem so we can have folks like Peter actively work with the lead committers on the kernel - to work through the queue.

For me a pre-requisite for being a kernel committer is a willingness to be responsible for reviewing and applying patches, etc.  Folks should not ask for Kernel commit just because it is convenient for them or because they have been around a long time or because they have strong opinions about the kernel.  Commit should mean commitment to doing the work needed - not just the work that is interesting or locally important.

By the way - if I apply the above criteria to myself - I would not give myself kernel commit - at least not from September through April when my time is 100% committed to being a professor.

I will be honest here - if someone is on the kernel commit team and they do not show the necessary "commitment", they should be removed as committers - they become committer emeritus - with great honor but not write access to the source tree.  Per Apache the emeritus thing happens after a while - but it is something the active commit team manages like anything else.  And folks can come off emeritus really easy if the active commit team so deems.

By being really Apache-like on this will make it so that whoever the active committers are, their jobs are easier.  They can meet, talk, use IRC, (of course do things in the open for transparency) whatever and get things done efficiently.  If there are only two people - going to do the work - then let those two get to it and don't make them drag a bunch of people who are "honorary committers" along.

Yes, this is a bit elitist - but we need some real commitment to reach the quality of product we need to get to - In particular part of K1 is to learn how to do K2 - K2 is when we will have a real kernel - but then this all should be second nature for us.

/Chuck

New Feature in SimpleLTI for Sakai - Note to Self

Add feature so that properties of the form

common.toolid.launch=String

Act as if they came from the LTI Proxy Tool Registration file with
"final" set to true. If these are set in the sakai.properties *and*
in the tool registation, the sakai.properties - wins.

For example:

sakai.simplelti.launch=http://simplelti.appspot.com/launch
sakai.simplelti.password=secret

When the IMS SimpleLTI tool is multiply registered, each common tool id
can do this separately.

Interesting Article about Dimensions of Hyena Cooperation

The following article is about hyenas - but it does show that things are never so simple in a collective organization.

http://news.msu.edu/story/5606/

Here are some excerpts:

"In a paper recently published in the journal Animal Behaviour, Smith, a student in MSU’s Department of Zoology, shows that while spotted hyenas know the value of living together in large, cooperative societies, they also realize that venturing on their own now and then to hunt for food is often the key to their survival."

...

“Although spotted hyenas are 20 percent more likely to capture prey with one or more members of their social group, cooperative hunting results in multiple new competitors showing up because former allies quickly turn into noisy competitors once the kill is made,” she said. “So it’s the individual, especially if he or she is low in the hierarchy, that suffers a cost for having group members at that prey.”

I think that the relationship between the individual and the collective happens in open source in general at the personal level and in organizations like Sakai where institutions are stakeholders, it also happens at the institutional level.

Reacting to Michael Feldstein's Blog Post about BlackBoard

Michael Feldstein had this excellent Blog post:

http://mfeldstein.com/connecting-blackboard-to-sakai-and-moodle/

It followed up this excellent blog post by Michael Korcuska:

http://sakaiblog.korcuska.net/2008/07/16/blackboard-sakai-connector/

I just had to comment in Michael F's blog - could not resist... Here is my comment...

Michael F. - I *love* this post - I also love Michael K's post as well. The dialog is getting to the real meat of the matter.

The Sakai Community and the Sakai Foundation are not one and the same. I sort-of like your suggestion that the Foundation not invest any *Foundation* resources in helping make something "non-open" happen. However, members of the community can do whatever they decide to do.

However I think that it would be wrong for the Foundation to make such a strong statement completely closing the door on any interaction because of the patent.

Back when I was the Sakai Executive Director, I felt that it was very important to engage potential stakeholders in the Sakai product and community regardless of their use of patents or the openness of their source. Because I felt that open source should be open - open source should be an example of how to win without being protectionist.

Under this principle that I gave to myself - I engaged in many discussions with a number of companies who are notorius for having and using patents and for keeping their source closed. And I even talked to them under NDA terms to help align their strategies w.r.t. Sakai internally before making external commitment statements.

These "bad" companies include: Oracle, Apple, Microsoft, IBM, and others. Also some of the academic members of Sakai hold nasty little patents - I think one of these academic patents is in the top 10 villains on the EFF patent busters effort - but we let Sakai resources benefit Universities that hold and exploit patents that cover our space.

My perspective was that it was not my place to just "good companies" and "bad companies". Some members of the Sakai Foundation Board and Sakai Community were often concerned when I initially engaged each of these companies as the ED. But overall after a while people relaxed and realized that nothing bad happened to us by talking to Oracle - that we effectively were protected because we were "open" and as long as Sakai stayed true to its own values - consorting with Oracle or Microsoft did not change our values.

And actually working with Oracle has had wonderful benefits - both to Sakai and to the market in general - your leadership in the IMS Learner Information Service - is awesome - Oracle's investment in a public good is much appreciated - at least by me.

I never look at corporate decisions as "good" or "evil" - they are either "smart" or "dumb". I think that Oracle's decision to engage in Sakai and IMS is "smart". I think that BlackBoard's engagement of open source is "smart". I think that BlackBoard's work in IMS Tools Interoperability is "smart". I think that the way BlackBoard granted patent immunity to Open Source is "smart". I think that the way BlackBoard pursued the D2L patent is "dumb". Not "evil" - not "good" - just "dumb".

I think that the market impact of the "dumb" BlackBoard decision has already caused negative consequences to BlackBoard - I think some of your earlier posts about dwindling market share are best explained as the results of management mistakes made by BlackBoard. I actually trust the market to punish "dumb" things and reward "smart" things - so my opinion really does not matter.

If I am asked - I always tell companies what I think the "smart" choice should be. I always preface my advice with the following variant of Sarbanes-Oxley: "Thanks for asking for my opinion on this matter, just so you know - I am just a humble academic with no real business experience, but if I were facing this decision, I would ...."

Apple iPhone (Original) Geotagging photos with iPhone 2.0 Software

This is what I know so far:

When you upgrade your original iPhone to the 2.0 software - you can do geotagging of photos.

When I pull the photos into iPhoto - and do CMD-I I see GPS coordinates. These are as right as they can be using triangulation/Wifi tricks - but they are at least *close*.

It seems that FLickr does not see the meta data yet - flick does not even know that the photo came from an iPhone.

But I am sure this will all work out.

I wish I knew a site that I could upload a photo and have it dump all the metadata.

Given that I don't have 3G in Lansing where I live and that Geotagging was my most important feature - it means that I don't have to rush right out to get the new iPhone - I will still get one eventually - I will just let the craziness die down a bit.

BlackBoard Sakai Connector - Inside Higher Education

I was interviewed for Inside Higher Education in regards to the BlackBoard/Sakai Connector:

http://www.insidehighered.com/news/2008/07/15/sakai

I added a comment to the article:

When dealing with a company - we should always assume that their purpose is to increase profit. Increasingly, companies realize that by creating *truly open* ecologies around their product - they make *more* money rather then less money. When customers perceive the market as "fair" generally overall investment increases across the board. When there is a real or perceived battle between open source and proprietary companies, many organizations with money "stand pat" and invest in neither until things settle down.

Part of making a truly open ecology is standards - I work with IMS (www.imsglobal.org) and we have been developing standards like IMS Learning Tools Interoperability and others that touch on these areas. Blackboard is participating in these efforts and we hope that much of the aspects of the connector will ultimately be covered by formal, published, and interoperable standards.

But it is a delicate balancing act - products and standards move at different paces - if we wait for a perfect standard - products will be held off forever. If we never build a product - we never reveal the fine technical details needed in the standard.

When I look at all of this, the balance tips towards being engaged in the effort and trying to influence the direction to ultimately improve the experience of teachers and learners regardless of what enterprise CMS system their school or campus happens to have installed.

More thoughts:

I think that we as a community (or at least a few of us) have a choice. We can either be part of this effort and affect it so as to produce the best possible outcome for everyone - or we can ignore it and hope it goes away.

If we choose to ignore it - BlackBoard will not stop - Sakai is open source with a commercial friendly license - Sakai's commercial partners already take Sakai modify Sakai and sell product / service around it. How is BlackBoard doing the same thing somehow different? If company *A* sells services and software and promotes Sakai use that is great and helps build the Sakai community - but if company *B* sells services and software and promotes Sakai use - that is bad and clearly straight from the Devil? At least we should be consistent.

Feel free as an individual or even a school to like or hate BlackBoard all you like - and perhaps part of the reason some folks like Sakai is because they hate BlackBoard - whatever gets you working - I am happy to see that. But please don't suggest that by being part of Sakai as an adopting institution or contributing developer - that somehow we all signed the "anti-BlackBoard" pact.

Sakai is about "freedom and choice" - Sakai is about win-win not win-lose - Sakai is about making teaching and learning better. Sakai is positive energy - not negative energy.

Think about the Apache web server - The Apache Foundation and the Apache Community does not think of WebSphere as the Devil. As a matter of fact revenues from WebSphere fund developers to work on the free Apache that we all know, love, and use. WebSphere and Apache HTTPD are not enemies - they are *alternatives*. Having commercial and open source products that are compatible and closely aligned - increases our freedom.

Open Source is there to make sure that we keep our freedom and alternatives - when an Open Source community decides that increasing market share at all costs is a high priority - then it has become a company that happens to release its source code - instead of an open source project. When this happens, the entire value system changes and frankly we then lose that magical quality that is open source.

I think that this is a good opportunity for the Sakai community to reflect a bit and perhaps think about our core values and the extent to which we all really understand what "open" means.

IMS Summit: “The Coming Changes in Learning: Creating New Architectures, Now!

On June 12 the U-M and IMS/Global Learning Consortium co-sponsored the symposium “The Coming Changes in Learning: Creating New Architectures, Now!” - 3 speakers and many panelists who addressed a range of issues related to Open Educational Resources learning architecture.

This was an AWSOME summit. I particularly recommend the John King Keynote speech about what makes learning happen - and the panel discussion of all the attorneys fighting about Fair Use. Of course I am on the Functionality Mashup panel - I warn you that my panel is not nearly as entertaining as the Fair Use panel.

http://inst-tech.engin.umich.edu/media/?sk=imsum08

The web site also demonstrates a pretty impressive lecture recording capability provided by the University of Michigan School of Engineering.

Great Question - Sakai Framework Future

Sonette Yzelle asked the following question on the Sakai Developer list.

Hi All,

I am a developer at UNISA. We have to decide what will be the best framework to use in future. We want to become more involved/contribute in Sakai.

We are using struts currently but must decide now if we are going to use RSF, VM, continue using struts or use something else. We don't want to build in unnecessary complication in our codes as we battle to get developers with the right skills and we had 3 resignation in the last 2 months. Questions that game up during our discussion were:
* Will struts be accepted by the Sakai Community?
* What is the general feeling about RSF in the Sakai Community? I found that some hate it and others love it, but I guess you get that with any programming language.
* Can any framework be used in Sakai as long as the tool work?

Do you have any suggestions to help us with our decision please?

-- Sonette

Sonette,

This is a tough question - there are many opinions - this is my opinion:

- Sakai supports a lot of frameworks - so you can choose - as you say - choice is frustrating and wonderful at the same time.

- If you are building something for UNISA only - I would go ahead and use Struts - it will be around for a long time and there is no reason to expect Struts support to become problematic in Sakai

- If you want to build a new tool with wide distribution of a tool across the Sakai community - RSF is the right choice of today - most new work intended for broad distribution I know of is being done in RSF.  I would advise against using either JSF or the Jetspeed-inspired Velocity pattern.  Neither of these comfortably supports the kinds of web 2.0 kind of stuff that will become increasingly important. In RSF you can gently move between UI stuff and Web 2 things like RSS feeds without hacking things.

- The sdata / widget approach is indeed the promise of the future - but it does not yet work with any shipping version of Sakai and will see its first production use in Cambridge in a few weeks.   I suggest letting it settle down in production for a while and see what comes out in Sakai 2.6 to make sure that the future direction for this technology is really worked out.   A key to this new approach is that tools end up with a complete redesign - assuming Ajax means that the UI for a complex and powerful tool looks much simpler - so tools end up being built (pretty quickly) from a fresh ground-up perspective.  In addition RSF tools will work in the new world in a backwards compatibility mode for a long time.

Of course the SOLO work is all GWT - which looks very awesome - other projects such as OpenSyllabus are using GWT - for now this is interesting and we will have to see how GWT in Sakai shakes out.

So, if you factor all the risks/probabilities - the safest path forward for now is probably Struts for internal UNISA stuff and RSF if you want to build a tool for the rest of us.  As you build new capabilities - make sure that you use a service oriented approach and separate your model from your controller/view -because you might end up making a widget to view the data in addition to the RSF tool.  If your APIs are clean - making the widget (or GWT version) will be much simpler than if you just mash the presentation and model code together.

Take a look Aaron Zeckowski's GenericDAO code for good API patterns:

https://source.sakaiproject.org/contrib/programmerscafe/genericdao/trunk/

Good APIs lead to good REST interfaces which lead to cool support in SOLO and easy to build widgets.

/Chuck

Progress on SAK-13584 - MailArchive Performance Improvements

I got the conversion working and all the code checked in. I used the Ian/Jim conversion gadget. Initially it looked like a crazy contraption. It turns out to be pretty cool - just undocumented - and the way Jim/Ian used it in Content was totally Ninja-warrior.

Luckily my new conversion code is a *much* better sample to work from - because it is simple. Some documentation should be written.

So it is now off to Tony for testing. More detail below.

Now I am off to go get my motorcycle gear from www.ridersdiscount.com. So for the rest of the day - call the cell.

The first version is up - primarily for testing - I only have a mysql convertor. It runs with mysql and hsql. Here are checkout, compile, and conversion instructions:

Grab the following branches:

https://source.sakaiproject.org/svn/db/branches/SAK-13584/
https://source.sakaiproject.org/svn/mailarchive/branches/SAK-13584/
https://source.sakaiproject.org/svn/util/branches/SAK-13584/

Compile and deploy

Run the conversion

cd mailarchive

Edit upgradeschema-mysql.config to get the database connection right.

Run the script:

sh mailarchive-runconversion.sh \
-j "/Users/csev/dev/sakai-trunk/apache-tomcat-5.5.23/common/lib/mysql-connector-java-5.1.6-bin.jar" \
-p "/Users/csev/dev/sakai-trunk/apache-tomcat-5.5.23/sakai/sakai.properties" \
upgradeschema-mysql.config

Check to see if the BODY, SUBJECT, and HTMLBODY columns look reasonable. The conversion can be stopped and started. If you want to undo the conversion simply execute these commands:

ALTER TABLE MAILARCHIVE_MESSAGE DROP COLUMN SUBJECT;
ALTER TABLE MAILARCHIVE_MESSAGE DROP COLUMN BODY;
ALTER TABLE MAILARCHIVE_MESSAGE DROP COLUMN HTMLBODY;

--Chuck

Test Scenario for this interim version - RUN this test and send me catalina.out

Here are the needed clicks - keep track of which ones are dog slow and which ones are fast. I expect that the non-search queries will be quite fast. And even the search queries should be tolerable.

Go to the mail tool in a site with lots of messages - do not enter a search value

Press next page 2 times
Press back page
Press last page
Press back page twice
Press first page
Enter a message
Press next message twice
Press back message once

Switch to Sort by Subject
Switch to Ascending and descending
Press back page
Press last page
Press back page twice
Press first page

Now put in a search string that is common and repeat the above steps.

Now put in a rare search string and repeat the above steps.

Send me back the catalina.out - thanks muchly

Mail Archive Tool - Improvements for 2.6 (SAK-13584)

Hot on the heels of my MailArchive Performance Improvements for 2.5 to handle large mail sites without chewing up memory and processor, I have pretty much completed an Alpha version of the next round of MailArchive performance improvements based on a database conversion extracting fields from the XML. I was hacking on this in Paris, Barcelona, Cambridge, on the plane to the US and on the bus to Lansing.

Here are the relevant JIRAs:

http://jira.sakaiproject.org/jira/browse/SAK-13584 (2.6 improvements)
http://jira.sakaiproject.org/jira/browse/SAK-11544 (2.5 improvements)

I will check it into a set of branches soon for some more extensive testing across more databases.

Here is a summary of the changes:

- Fully normalized the MAILARCHIVE_MESSAGE table - all fields are in columns now. The XML must stay for a while (more below) but it is not used for any search or selection.

- Added Aaron's GenericDAO concepts to org.sakaiproject.javax - with some improvements. Order and Restrictions are unchanged. Search has become Query. I added a searchString property to Query and added a few setters which were missing.

- Changed the BaseDbDouble ORM code to support the new genericDAO pattern when it sees a fully normalized table. When it sees a search, it uses a where clause instead of scanning all the data in Java. Also it understand the notion of search fields as well as order by fields. And from the 2.5 modification, paging is implicit throughout as well. Now the BaseDoubleDb can get either a count of messages or a list of messages using Search, Order, and Paging with a single DB query that returns *exactly* the right row set and nothing more.

- Changed the MailArchiveTool to use as little session state as possible. I moved some information into REST-based URLs. Also any heavywieght information is put into Context, never in session. This leaves the MailArchiveTool with a small session state with a few strings and integers - and everything left in Session should be trivially serializable.

This restores all functionality of the MailArchive tool regardless of the size of the message corpus. Each screen display is exactly one SQL statement to pull back *exactly* the required data searched and sorted in the database.

This makes it *REALLY* easy to come up with a really efficient webservice or sdata feed for Mail Archive - since you can search, order, and scope data above the API - you can produce *exactly* the JSON you need to implement a sweet dynamically scroll view of MailArchive where as you scroll off the bottom, the next 100 or so messages come from the server.

This also allows very efficient retrieval of messages for something like the SOLO offline Sakai client that Psybergate has produced with UNISA and NWU or other Google Gears views of a site's MailArchive Corpus. When SOLO meets SDATA at some future point in time - it might be a shot "heard round the world".

In general, I am pretty excited about the possibility of moving a legacy tool from being XML-based- to be very efficient and REST friendly. It was not that hard - once I fixed the DB layer to understand Query - I was mostly removing code from the Controller/View code (MailboxAction.java is much smaller now).

I really wanted to get the session to nearly zero and have Restful URLs for everything. But interestingly because the portlet model has this notion of Action and View being separated (often by a redirect on post), it makes it harder to put all of your state into the URL :(. So the compromise is a bit of session (as little as possible and all serializable) use REST urls as much as possible to let the back button work a little better - at least when you go back and click on a different button in a list view - you get the right message :). I need to do more study on this. JSR-168 has some features that make the back button more friendly than the old Jetspeed pattern.

The problem with the back button is really traced more to the Jetspeed 1.0 portlet pattern - it is not something inherent in using Velocity. And interestingly, this is another moment in time where my respect for the thought that went into JSR-168 grows.

I will begin checking this stuff in soon. I need to take a brain break and think about something else for a day or so and then review it before I make all my branches so folks can take a look.

TODO:

- Testing across all databases

- Think about a caching layer inside of MailArchiveService - it would only be worth it if there was searching happening on a very very large corpus of mail. But I don't want to optimize this without some tests on a large Corpus. (hint hint Tony)

- Implement the Restrictions from GenericDAO in BaseDbDouble - not just search - make sure to support both Restrictions and Support - or perhaps decide to do that later since MailArchive won't exercise it. Maybe just a warn for now.

- Make sure to refuse to search, restrict, or order on fields not supported - do this in DbMailArchiveService

- Work with Aaron to come up with a single world-view of GenericDAO - I hope he will like my improvements. I wish I could have had a beer - but we *just* missed each other in Cambridge this week.

FUTURE WORK:

I may look into doing this same exercise for the old Chat1 code - it should drop in pretty easily - Chat1 is even simpler than MailArchive. If folks are still using Chat1, I might be motivated to do it for 2.6. Let me know if you are interested in speeding up Chat1 and normalizing its data.

This paves the way for a whole new implementation of DbMailArchiveService - completely removing any dependance on the the DB ORM pattern and re-implement in either Generic DAO *or* perhaps Cayenne. When this is done, the XML column can be dropped completely. There is a little sticky wicket to think through - whether to drop the notion of accept() in Filters. I don't plan to do this until after 2.6 is done - just something for me to think about for a while.

Sakai Conference - Paris

Wow - I finally get a few minutes to blog about Paris. I had two talks which are listed at the end. Initially a few reflections.

This was our second European conference - I think that it worked well - we met many new faces - this is always important for open source. I wish we could have gone sometime other than the middle of summer - there were too many tourists in Paris :).

Probably the most exciting part of this conference was the participation of teachers. Thanks to Josh Baron of Marist - we had a great contest for the best use of Sakai with two wonderful winners who gave wonderful presentations. The coolest thing was the fact that half the room was filled with developers and tech support folks.

Usually we are told what is wrong or what needs fixing or what is hard to use about this software that we work so hard to make better. It was nice to hear from someone who *loved* using Sakai and felt that Sakai enabled them to teach better. I know that I could not teach the way I want to teach on any other learning management system - period. So while we are continuously bombarded about how we can improve (aka criticisim) - we need to remember that what we have here is pretty damn functional as we stand.

The other amazing thing was the amount of innovation *around* Sakai - the SOLO offline system using Google GWT and Gears was amazing, the IMS ePortfolio export from Edia was amazing, seeing folks working on the Open Syllabus in GWT was neat as well. Of course the MyCamTools stuff just gets better and better every time I see it.

I had spent some time working on McGraw-Hill's Katana before Paris and helped write their install documentation. I like the MHE approach because it gives teachers control over the organization and arrangement of the content and allows them to edit the content other than the learning modules. The large and complex learning modules are served from the MHE servers. The software is free, the content is free for teachers t explore - and only the students pay (like a book) when the teacher decides to use the content. Very cool. The Katana content is not part of a book - it is useful materials that can be used with any book - McGraw-Hill or otherwise. I want to get MHE to use IMS LTI as well so we can quickly get Katana Content into Blackboard and Moodle and other places as well.

The whole Michigan team had some good talks with Michael Feldstein and John Lewis about doing some early work with the IMS Learner Information Services spec. I would really like to see that happen.

The food was great - I went to the Eiffel Tower and Notre-Dame and not much else tourist wise.

I am really looking forward to the regional conferences for Sakai now that we have moved to an annual conference. I have a feeling that the Virginia Tech teaching with Sakai meeting will be awesome.

Here are my talks:

Teaching in the Open

http://www-personal.umich.edu/~csev/talks/2008/2008-06-30-teaching-open.pdf

Functionality Mashup - IMS Learning Tools Interoperability

http://www-personal.umich.edu/~csev/talks/2008/2008-07-03-lti-sakai.mp3
http://www-personal.umich.edu/~csev/talks/2008/2008-07-03-lti-sakai.ppt

More fun in Barcelona - Google Summer of Code Meeting

My good fortune continues. today I spent the day at Marc Alier's home on the north side of Barcelona near the beach. The first part of the day was spent talking and meeting everyone over a long lunch that included several kinds of Cava, snails, and Risotto.

Then Jordi and I spent about an hour designing the Moodle Module and Filter for the Simple LTI consumer tool. We talked about the structures and what additional features to add if we got done early. The basic outline was to work on the Launcher, the admin configuration of final configuration variables, virtual tools, a two-layer identity data sandbox mechanism, and a set of admin tracking, monitoring, white-list/black-list features under the control of the administrator.

Then Marc, Jordi, and I went to the beach on bikes. It was very nice and warm and we took a short swim. The cardinality of the set of female beach clothing configurations was one larger than on any other beach I had visited before - so that was an interesting experience.

When we got back we had Horchata and ice cream and talked about Texas Hold-em poker and how America might change as gas prices continue to increase.

P.S. Spain has already lowered their national speed limit to conserve fuel - making all unhappy - which might come to the US sooner than we think.

The Starfish and the Spider - The Unstoppable Power of Leaderless Organizations

I am like the luckiest guy in the world - after a great week in Paris with my second family of the Sakai community I was able to come to spend the weekend in Barcelona with some last minute arrangements and meet some new tech-geek friends.

After a long day of talking eating, drinking, talking and drawing architecture diagrams on napkins - we were going back to my hotel. Earlier I had told the group all about one of my favourite books - "The Innovators Dilemma".

http://www.claytonchristensen.com/publications/

On the drive to the hotel one of my new pals told me of one of his favourite books titled "The Starfish and the Spider - The Unstoppable Power of Leaderless Organizations".

http://www.starfishandspider.com/

Here is the short synopsis - the Spider is a hierarchical organization where each piece does its precise part in well-organized concert with the other parts of the spider. You can kill a Spider by putting a tiny pin through the Spider at that one place where it is all controlled.

The Starfish by contrast is a collective of pretty independent units. While none of the bits can stand alone - you can cut off a whole leg of a starfish and it will grow back - or if you chop a starfish in half you will end up with two starfishes. So while a starfish needs some minimal sized chunk to survive - it is pretty robust when it is injured and no single injury can kill the Starfish.

The book talks about how the Spanish invaders quickly conquered the Aztecs and Incas with 100 soldiers by figuring out the leader and killing the leadership (Spider killed by a tiny pin). However as they moved north, the encountered the Apaches. The Apaches were an open source project derived form the original NCSA httpd server. (Sorry - I slipped into the wrong story for a moment there).

The Apaches were a decentralized free society that operated in a collective manner but where participation in collective action was voluntary. So there was no real leader of the Apaches. When the invaders found someone who *looked like a leader* and killed that person - it was just the old medicine man - who was the spiritual leader- and while the tribe certainly must have missed the medicine man and hearing long stories about the ways of the ancient ones around the campfire - ultimately the loss of the medicine man did not stop the Apaches from organizing to wage war and chopping the Spaniards into small pieces.

After a while the Spaniards figured out that sneaking 100 soldiers into the middle of a big gathering of Apaches and killing one medicine man - just resulted in a lot of Spaniard pate'.

So how did the Apaches finally lose? According to the book, the United States much later turned the Starfish *into* a Spider. They took only a few of the Apaches - told them they were "special" - and gave these new special Apaches some cows. The ownership of a cow proved to be pretty cool - milk, baby cows for Veal Parmesan, Ruth Chris Filet Mignons cooked at 1800 degrees soaked in butter - all that. Feeding the whole tribe Ruth Chris steaks got one a lot of street cred that could be used later when decisions needed to be made.

The cows caused structure to appear in the society - and folks started keeping track of who was up and who was down - the Apaches were no longer equals in a loose, flexible, and dynamic collective - there was a class structure - and who you were and where you fit in the cow owner society started to matter and structure formed.

While the Starfish is generally pretty resilient in the face of obvious and seemingly damaging frontal attacks, it is fascinating how collective adaptive organizations (Starfish) can slowly be transformed into hierarchical static organizations (Spiders) by coming in the back door - intending to "help" and treating some members of the collective as "special".

In Paris doing some Code (SAK-13886)

Well - there is a performance issue in the portal code in Sakai 2.5. There are more queries to the SAKAI_SITE_PROPERTY in Sakai 2.5 than in 2.4.

In looking at things there were several instances of "too many queries" - not just from 2.5 - but for a few features that were added to portal between 2.3 and 2.5.

So I pored over the code for the last three days and figured out what kind of optimization which had already been done - and figure out where bits were leaking through the prior optimization.

I talked to Aaron and Ian - and came up with two solutions - one using memcache for the properties and another using thread local.

I think that this will be important for Sakai 2.5 sites - My guess is that in the summer many are not noticing the additional query load on the main portal displays. Without these patches things will likely slow down a bit and/or consumer more resources.

I initially made a property to control which tables the properties get cached in memory - my guess is that particularly for 64-bit JVM's this should be

DbFlatPropertiesCache=:all:

But at a minimum this is needed:

DbFlatPropertiesCache=:SAKAI_SITE_PROPERTY:

After some testing and experience we should figure out which of these two should be the default. I think that the methodology (suggested by Ian) is to look at query patterns and see if you are seeing too many hits on tables like:

SAKAI_SITE_PAGE_PROPERTY, SAKAI_SITE_TOOL_PROPERTY, SAKAI_SITE_GROUP_PROPERTY, ...

Then simply add them to the list in preferences - and see if things get better.

This is a nice, clean, short-lived memcache - and it is automated perfectly when properties are changed on the same server (i.e. when an instructor makes a change and presses refresh - the display will be perfect - instantly reflecting the changes).

The Thread Local code is obvious and must be done. It simply makes sure that the portal only gets the list of visitable sites once from the database.

Both of these mods are really simple and clean and should be trivial to review.

I currently have some print statements that can be uncommented if folks want to watch the new caches in action.

Once some testing happens - I will remove these before sending it to trunk.

Now this is done, I can start working on my talk for tomorrow about functionality mashup.

Here is my message to the dev list about it.

test - you can use the code in these branches:

https://source.sakaiproject.org/svn/db/branches/SAK-13886

https://source.sakaiproject.org/svn/portal/branches/SAK-13886

This is all described here:

http://bugs.sakaiproject.org/jira/browse/SAK-13886

Since this is a cache thing and thread local thing - I am happy for
folks to look it over before I move it to trunk.

Once this is done I expect to put it into 2-5-x - I would guess that
folks will want this performance fix to get the portal views less
costly in terms of the number of queries needed to produce the main
page.