Charlie Macchia asked: I

This week my oldest daughter went off to College at Michigan State University (my alma mater) – I stood in the doorway in my robe wishing her luck and hoping the day went well for her. She already graduated from high-school and has a two-year degree and full-time job – so it was a little weird to feel almost as nervous as the day she toddled off into Kindergarten.
After she left I was thinking about how she had chosen her school and the non-trivial part I played in her decision :). She got her two-year degree from Lansing Community College – I love LCC – I went to LCC and I have taught at LCC. LCC is a wonderful place for students to go after High School – it has structure in places and less structure in other places. The advisors and faculty are friendly and caring and the class sizes are small. There is inexpensive covered parking and many of the classes are in the building connected to the parking structure – all in all – a wonderful environment.
As she was finishing up her degree at LCC – the discussion turned to where she should continue her education. She had several places in mind – Ferris State University, Olivet College, and Western seemed like a good fit. She was particularly attracted to the 2+4 programs at Ferris – where they simply came to the LCC campus and taught the courses in the buildings she was already familiar with. Simple and effective – just keep going to the same, comfortable place, and a few years later – poof – out pops a four year degree – slick as a whistle.
She also had a high school friend who went straight to MSU and Amanda had heard horror stories about attending MSU – large classes – distracted instructors – long walks between buildings – expensive parking. So she really did not want to go to MSU – and in particular given the alternative of a few more years in the comfortable LCC building – why bother making it complex? Why take the risk?
But for me – even though I am fascinated with how we will learn “in the future” and the idea that you can take courses anywhere in the world – and do it all using technology at a distance. But somehow – all I could think of in the decision was bricks and mortar – and not just any bricks and mortar – I wanted her to go to the exact same bricks and mortar as I had gone to. Mom got her degree there as well and loved the place too. The physical location mattered – and to me it mattered a lot. The bricks and mortar were my single most important criteria – all of my discussion with Amanda was justifying *why* MSU was her best choice.
So she applied and got accepted to MSU (along with the other places she applied to) – she knew pretty much that once she was accepted to MSU – she was pretty much stuck going there – particularly if she ever wanted to have a conversation with me about college again :) She also knew that if she went to MSU she could get a Red Lobster dinner anytime she liked. She sold out to dad. I picked up her car payment while she goes to MSU. You get the picture… There are many rewards to be had by making dad happy…
So off she went on day one – a little excited and a little scared – and me a little excited and a little scared – you never want to force your child into doing something – they need to live their own life. I was afraid that it would go badly and then I would have been proven wrong – she would drop out and then go back to LCC and get a degree from Ferris.
So why the obsession with bricks and mortar? Probably because for me – going to college has had more impact on my life than anything else. I was the nerd picked last in High School – I had very few friends and never felt I fit in. In college at MSU – pretty much you needed to be smart to get in – so for the first time in my life – I met people like me – and I also met people like me that were more cool than me – so they showed me how I could be a more cool version of myself – without losing who I was.
Sure – I got a degree from MSU – but more importantly I got a personality – an approach to life and a set of life long friends – pretty much I grew up there.
I kept telling Amanda that the most important part of MSU was the people she would meet – the faculty and her fellow students – I told her that in many ways – you don’t pick a profession – it picks you. At a large university like MSU you can meet the future members of a profession and see if you like them. Frankly if you don’t like the kind of people who are in a profession – you should switch professions – and in college is the exact time to figure this out.
So what is necessary is some large and diverse population of bright people – all brought together around some bricks and mortar. Because the community that forms around the teaching and learning will have a far larger impact on your life than the actual material in a course – I know – I have taught college for years!
There is just no way to get this from a degree-in-a-box experience. The problem with a degree-in-the-box is that you are likely to start out pursuing the *completely wrong* degree – if you are stuck churning through the degree – you will never rub shoulders with someone who will show you where your rightful place in the world might be.
I came to MSU to be a Biology Teacher – in an honors Calculus Class I met a guy named Kirk Messmer – he said – “Chuck – you seem kind of bright – you might want to get a student consulting job in the computer center – it is where I work. It is fun and you don’t have to get dirty or clean dishes.” That changed my life forever in a wonderful way.
So as I watched my little girl driving off to school – I hoped that she would find her profession and that her profession would find her. When I wrote that first tuition check – at $1000 per class – it seemed like a bargain – given how much she was going to grow and how she would find her real profession and find her place in the real world as a result – the tuition was a small price to pay. I was paying for bricks and mortar and a learning community and a context – oh yes – some of the money went toward “learning”.
Luckily the first day went very well for her – her teachers were passionate and interesting and a bit nutty (no degree-in-a-box here) the students were interesting and by the end of the first day she was beginning to appreciate the value of something other than degree-in-a-box.
Lunch was a buzz of activity – long lines noise – kids from all over the world – finding their place in the learning community – hearing many languages (no degree-in-a-box here) and watching the Chinese students go around helping each other learn the ropes. In meeting the faculty and fellow students – she already had a strong visceral sense of what career in “K12 Education” is all about. She is excited – even the long walk was not so bad (I neglected to tell her about walking between classes in February).
So we got through the first day without her giving up and running back to the comfortable degree-in-a-box. Hopefully success will breed success and confidence will grow and she will stay at MSU long enough to get student hockey tickets that I can borrow if she does not want to use them!

There is a delightful debate on the dev list about grandfathering in folks with repo-wide commit into K1. Here is my thoughts – it certainly follows in the vein of my rant a few days back. But this post is more reasoned (i.e. less of a rant).
I am a person who ‘lost commit” to a lot of code that I care about, monitor, and work on regularly. I have not asked for commit nor to I intend to ask for commit for those areas in K1 – at least for now.
Given that Ian is very responsive now, and given that I do believe that changes to this most critical bit of our code base need more eyes and more thought than the rest of our code base – I think of not having commit as a *benefit* – not a problem. I wait 1-2 days usually – but as a result I always have a second set of eyes looking at my code and doing a bit of testing of my code and a smart person – looking over my shoulder.
Thinking back to how things worked when this code was spread across the repository and quite a few folks had broad commit – there are several problems that pop out – that are (for the moment) solved in K1:
– Many people got broad commit for some specific task and then we never took it back after the task was complete – often people agreed in the short term to not use their broad power without coordinating with whomever was the lead in the area – but over time they just decided that they knew enough about one or another area to just appint theselves as a committer and start fixing stuff on their own here and there. I remember a time when we would have a discussion about whether someone should have commit to authz – an usually the answer was “yes” – but now – so many people have commit on the whole repo that they get to make the decision about committing to authz them selves. And yes – we are lax about project leads – but frankly it is better to slow down and find a project lead on something than just declare open season with a bunch of repo-wide committers. SO the hypothesis that someone who had repo-wide commit should just get kernel-wide commit is a very bad one IMHO. Perhaps we should revoke a few folks repo-wide commit because their task is completed.
– Some people, (myself included) have commit everywhere because we are SVN administrators – but we never do anything other than what would be called “maintenance” unless we are really part of a project team – things like making branches, changing copyright, etc etc – are very different than fixing/improving code bits here and there. Those people with broad maintainer commit have been pretty good about knowing when not to use their commit. Again – I *technically* can trivially commit to Kernel – but have only done so to change the text of copyrights. Everything else – I go through the lead using the process the lead has asked folks to do.
– We have a number of different technical opinions about the overall direction of our code base and approach – we have competing architectural ideas and we have some pretty strong disagreements between some of our lead technical people. Far too often these conversations reach an “agree to disagree point” and when the conversation is between two committers with repo-wide powers – the two folks just start doing “both” things in the repo and trying to gain customers to “their way” of thinking over time – they get their particular approach into folks dependencies and then they can slowly win the day – frankly what happens is that we never get back to one solution – we just end up with two way of doing something for ever and lame compatibility between the two approaches over time. Our core architecture gets less and less clean as we have many approaches scattered across the SVN – each with a different level of maturity and different resources applied to them – and little cliques forming around approach X or approach Y – slowly splitting our thinking. Instead of working through the debate – folks with repo-wide commit just put the code in the repo and let us all figure it out. This is not a good approach to the most core bits of our code base.
I just think that it is a good idea for folks to have to justify their idea to a review team at a minimum before they just get to drop something into the kernel.
I guess the simplest formulation is this: Before K1 – if people disagreed with Ian on how something should happen – they could just say “piss-off” and commit the code that they wanted. After K1 – people who disagree with Ian do not have the alternative of just committing their opinions to the SVN and bypassing him.
So if you are advocating giving kernel-wide commit to all the folks with sakai-wide commit – what you are saying is that you want a continuation of anarchy in our approach to technical direction for the core code base – you are making sure we continue to be able to make major changes in technical direction with absolutely no review – if a person with svn wide commit feels strongly about some technical direction for Sakai in the morning – by evening – whether we know it or not, we are all stuck going in that direction.
I am sure that the folks who want this commit are the folks who have disagreed with Ian over the years – these folks are all smart folks and they have great ideas – and yes some chaotic organic improvement can be a good thing – but do you want chaotic improvement in the kernel??? Do we want the chaotic approach at this point in Sakai 2.x code base? Do you want to have alternative technical approaches explored in the code base and in your production instances? My feeling from the broad community (not the core 10 bright folks with great ideas) is that people want stability – not continuous change in our source tree.
This is where folks keep saying – “move the creative energy” to K2 – get involved – dig in – and fight things through with a group of people where we can try two alternatives as an experiment and then pick the best one without messing up production and a whole library of contrib developers – and a bunch of vendor branches.
The current K1 approaches insures that a technical disagreement will remain a disagreement until it has been discussed long enough and by enough people to make a community wide decision.
I am sure that folks will bring up the “I have a lot of bugs that need fixing” argument – for me that is not a good argument – bug fixing is *easily* done with patches – and a forced code review step is a great thing for bug fixing – and a great idea for the long term health of the code.
I know we want more unit tests – so perhaps the commit access to the unit test are should be opened up a bit – to folks who can make a long-term commitment to writing *and* maintaining all of the unit tests. This seems like a legitimate request – but getting commit in one area does not give one commit everywhere – folks can “ask for this commit” and when there seems to be long-term commitment to the code area in question – then commit can be granted.
Another excuse that folks want broad commit is “everything in Sakai’s current code base sucks – and I am here to find and fix all the suckiness wherever the suckiness is hiding – and given that I have a very small amount of time – I cannot afford to do anything other than committing the fixes to the sucky bits right to the code base”. This statement is wrong on its face – Sakai certainly has room for improvement in lots of places – but at the same time – what we have mostly works solidly and at scale. Those people who do not respect the value in the current code base – are *not* the people you want fixing it – because they don’t respect what is there – they don’t even bother to figure out what something does before “fixing it” – and then they will be off to the next sucky thing that pops into their mind and go “fix” that.
Everyone brings up “Apache” as the inspiration for this approach – actually this is somewhat un-Apache like. For simple projects like Pluto, generally when you are accepted as a committer – you get full commit to the whole source base. Common sense and community culture keeps you from doing stupid things in places you should not touch and as things evolve – perhaps you become the technical lead this time next year – you end up committing all over the place or even trigger a major refactor that you lead.
The reason that this works is that Apache has ways to resolve technical disagreements – strong disagreements are allowed in Apache and these disagreements are worked through in discussion to a resolution – these discussions are allowed to get heated and intense when necessary. If someone loses, they either get on board or (rarely) to fork the whole project (this happened in Pluto). When the person is so sure that they are right but lost the argument – they fork the whole code base – this way they are taking responsibility for *everything* – not just the little are they care about – this way they will learn the full impact of their decision before *anyone else* is affected. Some forks die – some forks live. My guess is that the most common case is that the person starts playing in the fork – realizes they took on a lot of responsibility and then comes grudgingly back into the fold – folks had choices – and with enough effort they could force their choice – and when their branch works – they are the new technical lead for a while – and everyone starts bitching at them :)
Our problem is that we never resolve the discussion and we never make a “fork” – the person who loses the argument – just uses their “svn-wide commit” to slide their alternative solution into the common code base – and all of our users, developers, and system admns are stuck with yet another confusing and inconsistent approach.
I have talked too long (as usual) – we need to slow things down in K1 – we need to stabilize the core bits of sour source base – the 2.X line needs to move toward “enterprise thinking”. Innovation and fixing the “perceived ills of Sakai” need to be done in K2.
Switching to K1 is an opportunity to bring a bit of discipline to ourselves that we are sadly lacking. Lets not throw that out immediately – lets see if we can make it work.
Interestingly this very argument is a place-holder for all those technical arguments – one group is advocating for a calm, conservative approach and the other group wants freedom to directly implement their ideas in the code base. As a developer – I like the idea of causing change through commit access – but as a community leader – trying to represent the broader stakeholder interests – I understand the need for limitations on my own actions – I understand that may I need some review before I go in and re-do some bit of kernel based on a brilliant idea I had in the shower this morning. And maybe the answer to my brilliant idea should be “no” – or perhaps the answer needs to be “next summer”.
P.S. If it gets to the point where Ian and his folk take more than about a week to apply my trivial patches – I will join the clamor to get commit for me. But for now with Ian responding super-fast – I am loving the new process.

I was digging through a pile of old stuff with a friend throwing away things which were no longer needed and came across the usability book Don’t Make Me Think written by Steve Krug in 2000. It was pretty thin and I had to wait for an oil change and some tires being installed so I started reading it.
I really like the book – it speaks to me and explains usability issues in a ways that make sense to me. Perhaps the reason I like it is that it was written in 2000 and my understanding of web technology is somewhere back in 1997-1998 so this is something I am capable of learning :)
Steve keeps it simple and gives some basic mantras that I find pretty easy to apply now when I am designing something. I particularly like that he does not try to make Usability something mysterious and complex and only best done by experts.
I now walk around and apply his “Don’t Make Me Think!” mantra all the time. Yesterday I was at Office Max buying some ink-jet refills. I knew my printer model and even knew the in “family” called “Epson Claria” – but when I got there, there seemed to be two cartridge series that were for my printer – so I wanted to verify which printer the cartridges were for.
Without thinking I picked up a box and looked at the back and sides of the box – looking for some fine print (black on white background) telling me which printers the cartridges applied. As Steve would say, we parse new situations using the rules we have learned form the old situations – when things “fit the mold” folks can use the new design *without thinking*.
When I did not find what I was looking for I was forced to think – yikes. Now I was in trouble. SO next I looked for a book of specifications – the other vendors had a book hanging at a particular location on the shelves – but for the Epson shelves – no book! Aargh – more thinking.
So next I saw the helpful sales person – she came over and I had to admit that I (A computer science PhD) could not figure out how to pick ink for an RX580! She took the box from my trembling hand and pointed out that the ink I was indeed suitable for the RX580.
The fine print was on the *top* of the box – it was small print – it was white on a blue background – and it violated the Steve Krug Home page rule – it was buried under two lines of “Happy Talk” (one of Steve’s rules is “Happy Talk must die”). It told me in English and French that there were five cartridges in the box – AAARGH – the box was 3.5 inches deep – OF COURSE it had five cartridges. I made a special trip to this store intending to buy five ink cartridges! AARGH. Another Steve rule is not to waste space repeating what the user already knows.
And it violated another Steve rule making the important data the largest font. The “how many cartridges” lines were larger font than the printer line. The printer list was in bold so they did get one thing right. The number 5 was very large and also in bold and it was the first thing you see when you look at the box top.
Of course my mind runs to the design meeting where they decided that putting the stuff on the top of the box was the best approach:
– It means folks can figure out which printers a box of ink applied to without touching the box
– And with the above improvement – we no longer need a box of ink cartridge applications – saving the company far more than the designer’s consulting fee
But – by making several simple mistakes, the decision made me think. Luckily I am capable of thinking and did successfully get my ink cartridges.
The right way to do it would be to put the cartridge applications both on the top and on the back in black-on-white print. Then I might have learned the new way – but also if I did it the old way – it would have worked. Another alternative would be to make the top of the box white and only put the printer list and none of the happy talk.
Now my mind is racing on the redesign of www.dr-chuck.com…
But the real upshot is that now I think of myself as a qualified UI expert because I read a book written in 2000 while getting my oil changed and tires changed.
Or at least I know enough to be dangerous to myself. I am looking forward to reading the whole book sometime. A transmission service should do the trick.

We had had our own little “mini-Michael-Phelps” event today when Brent won four gold medals at the Michigan state-wide 4-H show.
The state-wide show is where riders who do well in 4-H fairs and horse shows around the state during each summer are invited to compete at the state level with riders from around the state. The show is always held at the end of the summer at the Michigan State University Livestock Pavillion – it is a great event with everyone bringing their best game.
Brent has been riding as part of his physical therapy for Cerebral Palsy since he was five years old and he has been in 4-H since he was 11. He is now 17 years old so he has been riding for 12 years. Bonnie DePue (who runs CHUM Therapy) feels strongly that having therapy riders compete in 4-H is an important part of their therapy – it serves to motivate them to do their best and gives the participants a handicap-friendly event several times per year.
A bit of background:The 4-H has a special class for differently-abled children – it is called the Proud Equestrian Program (PEP). There are special classes for handicapped children to compete against one another – 4-H supports the PEP classes across all of its activities. They alter the activities to make them accessible for all. They do things like allow the PEP riders to ride first – in the cool of the morning – before the heat of the day as well as have specially trained judges for the classes. The classes are still quite competitive and riders are expected to perform flawlessly to the level of their ability to get the top ribbons. This is not just about participating – it is about competing – competing with yourself and striving to do your best in the ring with other riders.
Because the horse gives even very handicapped children mobility in a competitive situation, for many severely disabled children – 4-H PEP is likely the only time in their life where they will compete and earn an award in a physical competition. Children do not have to get a blue ribbon to be very pleased with their efforts and create great memories for their life.
Back to Brent’s story:Brent has been riding for six years in the PEP program and this was his last year in the PEP program – next year he moves into the regular 4-H program and will compete with all the other riders in 4-H.
So this year’s state show was his last competitive ride in PEP. Brent competed in four events:

• Walk-trot Equitation (unassisted)
• Horsemanship Challenge
• Pattern Riding
• Obstacle Course

The classes were pretty small – Brent put on his best riding performance ever (at least the best I have ever seen). He sat up tall and proud and hit every mark perfectly and did each of his gait transitions with careful focus. It did not hurt that Bonnie had let him ride Splash. Splash is my favorite horse of all-time. Splash belongs to Chris DePue (Bonnie’s son) – and Splash is very well trained in jumping and equestrian events – Splash is so good with the children and a great horse to ride (I always tell Splash that she is my favourite puppy – because she is so cuddly).
The announcer knew it was Brent’s final PEP ride and as the Gold medals racked up in each of the events, the announcer let the crowd know that this was Brent’s last PEP event ever – which made him even prouder of his work.
At the end, Brent had won four gold medals and won all of the classes. The judge/announcer told him to take trot Splash around the arena for a “victory lap” – which Brent did to a standing ovation and more than a few teary eyes.
The 4-H state show is very competitive and I am guessing that four gold medals is possibly the record for a single individual to win in a single state show. Like Michael Phelps – this may be something that will not be repeated. Brent was at the top of his game – at the end of a long and distinguished PEP riding career – the classes were small. Since it was PEP, there is an asterisk – But who cares!
Next year is Brent’s last year in 4-H – he is a senior. The senior year culminates with a special ride at the Ingham County Fair called the “senior ride” where the rider walks around the arena while a short story is read about what they have done in riding and what it means to them. If you come – bring tissues.
By the way – you are always welcome to donate to Bonnie DePue’s – CHUM Therapeutic Riding at www.chumtherapy.net. CHUM Therapy is a non-profit organization that is dedicated to making this world a better place through horses interacting with handicapped children.
Bonnie understands that the purpose of CHUM is far more than one hour of physical therapy per week for these children – and I publicly thank her as often as I can.
You can see some online videos of CHUM Therapeutic Riding that I have produced over the years.
A video about Brent.

Note to self – remember this code sequence from Aaron:
You can normally do something like this from just about anywhere:
try {
InetAddress i4 = Inet4Address.getLocalHost();
String ipAddress = i4.getHostAddress()); // IP address
} catch (UnknownHostException e) {
// could not get address, do nothing?
}
-AZ

Sorry – this is a rant. The rule in rants is that you don’t have to be logical – you are just mad. So I apologize in advance… I don’t intend for this to be about anyone in particular but there are some example things that I call out – just because I call something out – it does not mean that it is a bad idea – and certainly does not mean that I am upset at a person who is just trying to do what they think is right. I am ranting about governance and process and approach – and community culture – not any single individual.
Back in 2004 and 2005 we had a “chief architect” – that was either me or Glenn – or most likely a combination of both.
The major role of the chief architect was to worry about the the entire scope of the Sakai release and not look at just their little corner of the code base and be “selfish”. Having global conventions and rules that were enforced across the code base – make everyone’s life better. And to really *have* global rules – you need policing activity and judges and you need to tell some people at times to “stop that!”.
Now a few people in the community over time looked at this police/judge (chief architect) role as a great privilege and a glorious source of ego building power. Actually this role is one of the crappiest roles in any project if it is done well. Everyone is pissed at you all the time and sometimes a person who is told “no” they go find a few friends and come at you with a mob when they get really mad. Sometimes the chief architect is wrong because they needed to be educated and has to publicly admit it and reverse their position. You cannot survive the chief architect position and keep your ego – it is beaten out of you if you do the job right.
—Sub-Rant—-
The key in any open source project is that leadership is action – not position. People who come from organizations where leadership is not valued and instead management is the more valued than leadership – they react to titles and positions differently. People who are sadly stuck in and/or grew up in in manager-oriented organizations get all veklempt about titles. Those who have seen leadership truly modeled really hardly care less about the title and focus on what a person *does* – in organizations that value leadership – management activity is seen as another task that is equal in importance (and often less rewarding than) non-management work. Management generally is the act of “taking care” of the non-management people – and since this is tiresome – it is often well-compensated. But in truly leadership-centered organizations – having a title of “manager” or “director” does not automatically make you better than other people in the organization – you must earn your rank in the pecking order based on what you *do* and how well you do it – not the fact that you have a fancy title.
So, these people from unenlightened organizations see the chief architect role as something that has a lot of power and little responsibility – and since they have never experienced nor understand real leadership – they treat that person like all of the useless troll managers they have met in their life and either whine and complain or kiss butt / play politics to get what they want.
—-End-Of-Sub-Rant—-
Back to the original rant.. So the chief architect role is an unloved and unappreciated role – they make decisions and tell people “no”. But here is the key – they also take responsibility for the *whole code base* and the whole release process and after release process and the production issues and and and – if something drops through the cracks – this person picks it up – or at least tries to catch things and warn folks – or get the right folks involved.
We don’t have a chief architect any more as best I can tell. I am certainly not it – and I am glad – it was a role that kept me nervous all the time and I was glad that while I had that responsibility I was paid full time with unlimited travel budget to do nothing other than guide the Sakai community – it is so engrossing – that it does not blend well with other tasks.
It is not clear exactly why we don’t have a chief architect role any more – the two biggest contributing factors are (a) management types wanted control of the core decision making processes – so that decisions would be made manager-style (lots of meetings, lots of politics, lots of quid-pro-quo, lots of compromise, and none of those loud technical people involved in the process) and (b) because we just got too big for one person to know and watch it all. The reality is that it is a little of both.
However we got here – we no longer have one-stop shopping in terms of when things break – who do we ask and/or blame. We make technical decisions Murder on the Orient Express style. The deed is done – and there is no one to blame because everyone naturally takes such “fractional” responsibility.
The way this works out in Sakai is that someone proposes some broad new approach in a meeting or on the list. A few people give a +1 affirmation or grunt assent in a meeting – the person walks out of the interaction claiming they have a “mandate” and proceeds to take us down this new direction.
Often the proposals are called “improve our XX approach to be in line with industry best practices” – and then this gives folks a carte blanche to change our overall direction in some small way. So they go off and change our overall direction – and if challenged – they say something like “the folks in the meeting at XXX seemed to think this was OK”.
Usually I am the one opposing things – and usually I am the lone opposition voice (or at least the only one who has the guts to try to say “no” out loud). And so it looks like the “people advocating moving towards best practice” (apple pie, patriotism, world peace) versus the old cranky guy.
I am vastly hampered in my attempts to be critical about dangerous changes in direction is because the people I am arguing with are all smarter than me technically – and I know that. The top 20 people in Sakai are so damn smart it is not funny – they are all way better developers than me. But they are also often young and in a hurry and do not have experience with the long-term pain of seemingly simple decisions.
As an example of long-term pain, in late 2003 – as chief architect I decided that Java Server Faces was a good UI technology. It turned out to be ABSOLUTE CRAP! The effect of my decision was to cause great harm to this community and my estimate of the wasted effort from that single decision to be about 2-3 million dollars across Sakai – it doomed several projects to lose years of progress – particularly Samigo – I feel horrible about the decision to rewrite Samigo in JSF. And the new chat is in JSF – AARGH. I live with this decision and publicly admit my shame on this over and over.
The key lesson is not that somehow I am stupid and all the other people in Sakai are smart. The key is that sooner or later a mistake will be made and that a bit of caution is a good thing – and most importantly – the pain of a decision is often not evident for months or years after the decision was made.
We are making decisions now of the same magnitude pretty regularly – and we won’t know whether they are right or wrong for a while. I have opposed both of these vehemently – and folks just treat me as the “old cranky guy”.
– The use of Wicket in the main SVN and Sakai 2.6 is absolute crap – it comes up with ugly warning messages about Wicket configuration and this is ta test version…. Aargh.
– The independent versioning of every little piece of Sakai – I call these “vanity versions” – is absolute crap
Perhaps “absolute crap” is to strong of a term here – but at least lets *delay* these decisions and explore them in experiments with production systems before committing the whole community and whole release to go in the direction.
But NOOOO – someone got three “+1” votes and one “-1” vote from Chuck – and we all discount Chuck’s vote to be “-0.001” because he is just an old cranky guy who says “no” to everything.
I am an old cranky guy because I know that we are making these decisions too quickly and with no real consideration of the long-term consequences three years from now. We have no chief architect and we have no structure that moves us toward being conservative in the overall direction of the product. Our governance structure lets major shifts in direction to happen with the agreement of 3-1 vote out of 100 developers.
Another thing that annoys me – is that we *do* have smart conservative folks in the community – other “cranky people” who are generally nervous about too much change – too fast. But they are also *verrry quiet* people.
In a few years – the code will be a swamp. Making a release will be like finding a needle in a haystack.
Now – all is not so dark. Here are a few really brilliant points of light in what we are doing:
– The Kernel 1 work is brilliant – it took over a year to make sure that this was a good idea – Ian gave us many opportunities to look at his idea and he patiently did it over and over again each time someone raised a question or issue. He did his work as a branch and let it become nice and boring before we switched to his new practice. And the switch was awesome and pretty much flawless. Even with this much preparation there were a few things that needed thinking through after the switch – but with Ian as our leader – we could work through decisions while letting Ian keep the big picture intact.
– Take a look at this JIRA – http://jira.sakaiproject.org/jira/browse/SAK-11875 – Look at the reasoned conversation – look at the depth of thinking look at the attention to detail – and most importantly look at how many really bright people brought their thinking to the problem and how deeply they thought about the problem. We are blessed to have 10-15 people who are smart enough to see the whole code base and work collaboratively to narrow down a really hard problem and produce a superb solution to the problem. So we have 10-15 people who have talent to be architects.
But we still have no chief architect – we still do not have someone who feels pain when the release breaks – like in the Orient Express – when the release tanks – no one feels responsible for more than a small part of the release – and the cause of the failure is likely to be someone else. The math is simple – if you own 5% of something and the whole thing breaks – there is a 95% chance that it is not your fault. Except of course for me – I screw up more than most so even though I touch about 5% of the code – if something screws up – it is 10% chance that it was my fault :).
So we have these people that make long term decisions – fight fiercely over my lone opposition and win – drop in their change and then run for the hills leaving the community to do the long term support of their new decision – and the sad bit is that 2-3 years later when the overall impact of the decision comes home to roost – the people who screwed it up will likely be gone and so there won’t even be the satisfaction of “I told you so”.
This is all OK – open source communities have people that come and go and new folks pick up from old folks – this works except when there are major direction shifts or major changes in community commitment that slip in under these “+3” votes.
Enough whining Chuck! What is the solution?
A year ago – I yearned to go back to the Chief Architect Model. I did not want (and do not want) to be the chief architect – I just wanted there to be one. I wanted to have the feeling that someone was watching over the work that we all do – and would swoop in and save us from ourselves. It is nice to know someone has your back. We kind of have that for the Kernel in Ian – this is cool – I want to help and cooperate with Ian and I hope we all support Ian and help him – he volunteered for a really painful job. Don’t look at this as a “power grab” – look at as a “life sentence” that he voluntarily accepted – be thankful and appreciative – send Ian a “thank you note” once in a while – buy him a beer when you have a chance.
What do we do in the rest of the SVN where most of the current crap is happening?
First, we clearly have enough talent to do this together and if we focus on the “one person” approach we leave a lot of talent on the table unused and slow things down to a crawl – and we also suffer from the fact that no matter how smart one person is – they have blind spots and they have personality that creeps into their decisions and pre-conceived notions that creep into their decisions over time – so many eyes on the problem of the size of the main SVN are a good thing. And it cannot be Ian – he is used up on the Kernel now.
Here are my recommendations:
– Bringing new technology in the core – like Wicket – needs to be done with at least one semester of production experience with software at more than one site. If there are folks who think wicket is the “next best bright shiny object” – let them prove it and let them vote with their production instead of the very inexpensive “+1” in an E-Mail. Wicket should not go into 2.6 – let a few adventurous sites run it and see if wicket has memory leaks or the internationalization sucks or it fails on IE7 (or whatever) The vote that matters is the “who ran this in production” vote – not the vacuous “+1” votes on the dev list. People need to *Stop* thinking about this as an ego issue. I am writing lots of code that is new stuff in Sakai – doing this in trunk of the main release is scary for me – my IMS tools will be run at UM for a whole semester before I would even make the slighted *peep* about them in the main release – this makes my life *better* not worse. It means that when they go into 2.7 – I can sleep at night knowing that I won’t be the one who crashed Rutgers production because I pushed my stuff into the main release before it was production hardened.
– When people want to do something that will make the “release process better” – like this crazy idea that we need 40 versions of things in the main SVN instead of one. Maybe this *is* a great idea – if it is a great idea – lets try it one place and get through a release and see if the process *IS* better. And then see if we can actually make a release with the “glorious” new ideas in place. If we do it only a little bit and it turns out to be a horrible failure then we back out the little bit. If we convert the whole SVN to the new approach and find out half way through QA that it was a bad idea then we have a massive back conversation. And folks – good ideas have staying power. If it is a good idea – it will happen in the next release – folks don’t forget about it – if the simple use of the new approach shows the glorious way forward – people will rush to the new approach. And maybe these folks who are so “smart” about how to change how we do the release – perhaps they ought to try to *do* a release with their proposed ideas before they just check it into trunk. There is a long way between Sakai compiling in a developer environment and Sakai having 100% squeaky clean release artifacts.
We can make these decisions collectively if we just do things more cautiously and folks who advocate new large-scale directions are willing to be patient. Frankly the last thing we need is someone who wants to make some global change *right now* because they want to get it done and then move back to some other job. This is a major red flag for me – it means that when their great idea takes a crap – they will be busy on something else. Making folks wait also tests their own conviction that their ideas is really good.
OK – so now my coffee cup this morning is empty so it is time to wrap up the rant.
All I am saying is give peace a chance. Actually no – his is what I am saying:
– Quit thinking that +3 is a mandate and sufficient justification for major change – the justification for major change is *proof* that it is a good idea.
– Take a conservative approach to new directions and new technologies – make them stand the test of time for 6 months before we dive forward – it is hard to go back
– Lets make a culture that makes it OK to oppose something. While I am the guy who publically does a “-1” on stuff – I do it so often that my credibility is shot – at least it is shot enough that my -1 alone means pretty much nothing when the proposer knows they are smarter than me (a common occurrence).
We have a lot of people with a long history of experience watching decisions happen and watching things hurt later because of those decisions (and yes most of the mistakes were mine) – but even though it is different people making the decisions – the mistakes are and will be the same – the core reason I made a mistake was when I was in a hurry and was not willing to try something out in the small before it because our “direction”.
Lets learn the “red flags”.
– A major rewrite that surprises the community right before code freeze
– Something that claims that it will *improve the release process* from someone who has never done a release
– A major new technology that one tool writer “discovered” and decided to use
– Anything where someone seems to be “in a hurry” or trying to slip in under the deadline
Change is essential – if we stop moving we start dying. All the decisions that have come back to haunt me over the past five years were “yes” decisions. It is easy to say “yes” – it is harder to say “no”. Just saying “no” is also a bad approach – no can be wrong as well.
We can recover far more gracefully from a wrong “no” decision than a wrong “yes” decision. If a “no” decision turns out to be wrong – we just turn it into a “yes” later. We hold on to the status quo a little longer – for those of us in this for the long term – we can be patient.
A bad “no” decision means we endured a “stable but sub-optimal” situation for a little while longer. A bad “yes” decision can cost us millions of dollars in wasted time and effort.
If a developer or team does not like being told “no” and want to make their own decisions – or thinks that Sakai is too conservative about changes to its release process or new technologies – and prefers a “more agile” approach – then just move your to contrib. You have 100% freedom in contrib – no one will tell you what, when or how to do something. People can put your code in production and if it craps out – then it is *your* creditability that drops and not Sakai’s credibility.
What I am looking for that will tell me that our process is healthy is when we as the community learn say “no” to something once in a while. Or at least we as a community make a strong statement “Please slow down” – to *something*. Once we as a community can learn to say “no” once in a while – maybe we can get to the point where our collective wisdom is better than a single individual making long-term decisions.
At this point my assessment is that we are worse off in terms of making long term decisions about the product than we were in the 2.3 era. From the 2.4 release and onward – we switched from the “chief architect” model to the “collective wisdom” model. We cannot and should not go “back” because one person cannot hold this all in their mind – but we do need to go “forward” to a point where we are truly reflecting on choices and making hard calls – and firmly telling folks “no” or “slow down” in certain situations.
Until we do that – our code releases will get mushier and mushier and the maintenance branches will become more and more important. This is an indication that our process for managing trunk is not working as well – and we fix things when we have a branch-manager (chief architect for the branch).
I would like to see us making better decisions in trunk – and making the branches – truly maintenance.
End of rant.
One of my favourite sayings is that “A project is doomed if the smartest developer is the chief architect”. Having someone as chief architect who is smart but not the smartest sets up a natural balance-of-power and leads to well reasoned decisions.

I must be hacking too much. Last night I had a very vivid dream about the Google App Engine.
In my dream I am reading my E-Mail and I get a note from the Google App Engine folks that I have had too much traffic to my simplelti.appspot.com site and that now I had to start paying for it. I was all freaked out because the site was down and I could not figure out how to pay for App Engine (I don’t think it is possible yet – they just shut you off).
So this was really disturbing because I could not figure out how to get it back up.
One of the first things this morning was to update the site to add some screen shots of dokeos (http://www.dokeos.com/) supporting SimpleLTI (thanks Daniel) to the site. And after I finished uploading to the site – I realized that it was only a bad dream – whew.
It was a pretty vivid dream – the visual images of E-Mail reading in the dream were very accurate. This was part of why it seemed so real.

If you had an App Engine update go awry, you may need to roll it back. You will get a message like this when things are messed up and you try to “Deploy Application”:

*** Running appfg.py with the following flags:
--no_cookies --email=acct@gmail.com --passin update
Password for acct@gmail.com: Scanning files on local disk.
Initiating update.
2008-08-12 08:51:32,117 ERROR appcfg.py:1334 An unexpected error occurred. Aborting.
Error 409: --- begin server output ---
Another transaction by user acct is already in progress for this app and major version. That user can undo the transaction with appcfg.py's "rollback" command.
--- end server output ---
If deploy fails you might need to 'rollback' manually.
The "Make Symlinks..." menu option can help with command-line work.
*** appcfg.py has finished with exit code 0 ***

The solution (paths from a mac) is to running the command:

python /usr/local/bin/appcfg.py -verbose --no_cookies --email=acct@gmail.com --passin rollback simplelti

“simplelti” is the application directory – you need to run this where your application is a subdirectory.

simplelti.org_title=Univerity of Michigan (CTools)
simplelti.org_name=CTools
simplelti.org_url=http://ctools.umich.edu
simplelti.org_id=umich.edu
simplelti.org_secret=lastditch
simplelti.org_secret.appspot.com=notsecret
simplelti.org_secret.simplelti.appspot.com=secret