I was giving a demo at the Apple Academix Conference during a keynote where I use a Google App Engine application to simulate a clicker-like activity in a course. It is a simple multi-user number guessing game. Whenever the audience participation was too good and too many people started clicking on the application I would get errors – it made the demo in the middle of my talk lose a bit of credibility – but it was also funny.

I finally saw the error and spent a few hours to track the issue down. I was getting the following message:

TransactionFailedError: too much contention on these datastore
entities. please try again.

I had been using get_or_insert() pretty much with reckless abandon and wondered if it was smart enough to optimistically do a get() and see if the object was there before starting a transaction to do the get() and then put() if necessary. But given the behavior I was seeing – it looked like it was starting a transaction every time.

I had 20 participants and after the first click in the application, there were no new objects in the application and 20 people clicking within a 10 second interval were causing the error to occur. I suspect but cannot confirm that at some level the AE framework marks you as abusing transactions and then it takes a while before load drops down that it lets you once again use transactions.

So I decided to reduce my transaction abuse and build an optimistic get or insert that does the get() first and then if the record does not exist do the get_or_insert(). This way once objects start to exist there will be no more transactions. And yes, my next optimization is to use memcache.

So this was my approach.

Continue reading →

I have a Macintosh Air (first revision) that I purchased the day they were announced. I loved the unit – my only complaint was that the disk was a bit small. But the rest of it was so nice – it has been by my side constantly from the time I purchased it.
About a 2 weeks ago, it developed the dreaded “Dying Hinge Problem” – so I started to search the Internet and found that Apple is categorically refusing to repair these problems claiming that it is “abuse” and as such not covered under AppleCare. There are many vitriolic blog posts and a cool YouTube video where a guy stabs his Air with a knife (really) to kill it and then switches to an IBM ThinkPad.
I have been traveling since the problem appeared so I really could not deal with the problem until I got back. Whenever I was a bit bored I would Google more and kept coming to the same conclusion – I was screwed. Not only was I screwed – but I was going to have to have a bunch of really frustrating conversations with Apple as I tried to get the defect repaired – I kept running these imagined conversations over and over in my head in the Ann Arbor Apple store – and over and over I lost the conversation after much frustration and wasted time where I was mean to the employee and then went to the manager and shouted at the manager. I imagined making a fuss in the store to embarrass them into fixing their defect for free and failing to get my Air fixed – instead I just looked like a fool.
Now let me just say that this is a defect – the box was not dropped – I did not use my MacBook to shovel snow or remove roof shingles. Often my laptops get dropped once (and survive) but this one never was dropped. I have had 4 Apple Laptops over the past six years (and loved them all). Every single laptop ended up with a dead hinge except one. The mean time to hinge wear is 3-5 years – it just happens. It is *so* consistent that the only reasonable assumption is that the failure is engineered in to make sure that laptops don’t last forever – forcing us to upgrade. The only mistake in the first generation Mac Airs is that it gives up a year *before* AppleCare expires and not a year *after* AppleCare expires. A slight engineering mistake – best solved by simply telling everyone that if it failed before the designed-in 3.5 year failure period – then it *must* be abuse! So Sayeth Apple and we must obey!
Back to my story. I got back from England yesterday and was ready to start my battle with Apple – I decided to start online so I went to Apple Support online and typed in my serial number. And then I received the wonderful news that my laptop was not covered under warranty!
This made perfect sense – I had not purchased AppleCare when I bought the box (the $1799 edition) and thought to myself – “What might go wrong with this box that Apple would agree to fix?” Now that Apple insists on their service centers fixing things it is easier to simply characterize anything as abuse if they don’t want to fix it.
So when I calculated the things that Apple would fix and multiplied it by the probability that one of those (very few things) would go wrong and multiplied it by $1800 – I simply decided to keep my $250 and apply it to my next computer.
Damn – I was right – I am a genius! I am now sitting on a broken Mac Air with a defect that the Apple Borg would declare as abuse and waste *hours* of my time arguing about after which I would invariably lose anyways.
Not only did I save money with this decision – I saved time and peace of mind – an all around win! I do need a new laptop now because the Hinge problem (a defect) will just get worse – so I need to put this Mac Air into light duty.
Of course because of my wisdom, I keep my $250 and apply it to my next Apple purchase. I still love Apples and still will buy an Air as a replacement – the Rev 2 systems are faster with a larger hard drive anyways and I might just get a 13″ Aluminum MacBook anyways – a little more weight and a DVD and the ability to plug in audio cables without going insane and $400 cheaper and a larger hard drive and and and… Hmmm.
So here is my general recommendation to folks – keep buying Apples – if you get mad at Apple when they choose never to fix things and blame you – don’t switch to PC – just buy your laptops differently from Apple: (a) Buy the bottom of the line that you can tolerate – don’t buy that super-duper system and (b) don’t buy AppleCare! Perhaps if enough of us do that – they will get the picture.
This way you upgrade almost twice as often for the same price. Also those broken hinge computers make good servers or even desktop systems – you just need to stop hauling them around and opening and closing them. They last for *years* with the broken hinge.
P.S. The only computer of the four Mac laptops I have purchased that has *not* developed the hinge problem was the MacBook pro that was only 1 year old when I switched to the Air. Since then it mostly sits on a desk as my desktop.
P.P.S. The sad thing here is that this *is* a defect – it is exacerbated the more you use the system – lots of travel lots of security checks – working in airplanes – working in airports – working on a bus – make a laptop’s life rough. In a sense because the Air is so wonderful – we use the living crap out of them. It is sad that Apple can’t put a little more engineering into these boxes (hinges in particular) to support heavy use to insure these systems do last at least 3 years under heavy use conditions.
P.P.S. I am amazed at how solid my first-generation iPhone is. Prior to having the iPhone Generation-1 I had a series of Treo’s that never made it past 10 months before it looked like they had been through a war (I travel a lot). The iPhone gen-1 is over two years and there is almost no wear at all – very impressive engineering. Perhaps Apple should just make their next generation of laptops “hinge-free” and have a slide-out keyboard or on screen keyboard (or both!)

Fresh off getting my first Sling Application finished last night – I came in and told Ian that I had nothing to do today. Of course he said “no” – I needed to build a widget for LTI as well. I said “what widget” and he said that the whole Slikai UI was there under http://localhost:8080/dev and I needed to write a Widget.

I started work at 10AM – with a quick tutorial from Nico and his HelloWorld example, I hacked away at it. After about 2 hours working by myself I had it doing settings and creating the resource. At that point Nico helped me do the actual launch iframe and a few nice bits like a cancel button and it was done and put some polish on it.

It was done three hours after I started – and this was my first widget since I tried back in July 2008. I was pretty pleased with the ease of figuring things out – maybe I just knew more about AJAX, jQuery, JavaScript, and FireBug after writing those chapters in the book. However and whatever the case – it came together quickly and the ability to use Sling to handle all my persistence needs instead of writing a service and plugging it into sdata made an amazing difference in ease of building my widget and getting it running.

So now I am done with today’s task. All is now checked into my SVN repo in contrib/csev/trunk/sling

So I told Ian that I was done with the widget and still had the whole afternoon free! So he told me to add Pluto 2.0 (JSR-286) to Sling/Sakai. That sounds like a good idea – but more than a three hour tour. So we will just draw pictures and call it good.

In summary, I continue to be pleased with the Sling/Sakai combination and eagerly look forward to its evolution.

The additional README bits are below. Again many thanks to Ian and Nico for their patient mentoring.

Continue reading →

I visited Cambridge this week to see what was up with the new Sling + Sakai (Slikai) work that Ian is working on. I figured that one way or another I would find a way to hack in IMS Simple Learning Tools Interoperability into it – after all I hack LTI into pretty much anything else. One day I will write a FORTRAN version of Simple LTI :).

I figured that in two days I would be a little pressed for time. Day one started out with Ian explaining Sling for about an hour and giving me some hello world stuff. It took me a while to hack each piece together. I started this morning knowning nothing about Sling and now this evening I have my first working application in Sling/Sakai.

Sitting 3 feet away from Ian meant I was never stuck for too long – I was just slow. But by the end of the first day, I am pretty much done – the code is checked in here:

https://source.sakaiproject.org/contrib/csev/trunk/sling/

All in all I am pretty impressed with Sling. Many important decisions are already made – most of the APIs we think of as “kernel” are in place. There is lots of value to be added – mostly in terms of “helper” APIs to lock down “Sakai conventions” within Sling and keep us from hard-coding convention too widely scattered throughout the code.

There is lots to do – so much more to do – but Sling feels like a great starting point. Performance is always something to worry about – but I am confident that Ian worries about performance even more than I do – so I expect that whatever we run into is solvable.

It was great fun to see this at an early state and to talk issues over with Ian and I look forward to the progress over the next few months and hope that we see some nice shiny stuff by the conference.

The README is below if you want to replicate my experiences.

Continue reading →

This abstract was submitted to the OpenEd conference:
http://openedconference.org/
August 12 – 14, 2009 – In Vancouver, BC
We will describe our experiences supporting self-regulated learning through newly developed outcomes-based open learning tools at the University of Michigan Medical School.

Continue reading →

Here is a cool post about getting Rails (grails) running on AppEngine under Java.
http://blog.inigral.com/2009/05/07/rails-on-app-engine-a-cold-start/
This shows the vast potential of the Java AppEngine Java environment – because the JVM is really just a virtual computer – it can do anything in any language. The only consequence might be performance – but over time any linear performance factor fades away as a real issue.
Thanks for the link Michael.

Thanks to John F for sending me in this direction:
http://ocwblog.org/2008/09/28/byus-loosely-coupled-gradebook/
Some Specifications about the Loosely Coupled Gradebook
An overall vision of a loosely coupled learning network
Jon uses the Spider and Starfish Analogy and applies it to teaching and learning – wonderful post. In open source and open thinking – this book should be required reading and there should be a quiz that people must pass with a score of 100% before they can join an open source project or even use the word “open” in a sentence in public.
Time to do some research….

This is a summary of comments I made in a panel discussion on Service Oriented Architectures at the Open iWorld at the IMS Learning Impact meeting.
Here are the slides that I used in the presentation.
Colin Smythe spoke before me and mad a number of great comments that suggested that SOA had been overhyped and was now in a bit of the “trough of disillusionment”.
I started my comments by suggesting that SOA (Upper Case) was pretty much dead and pretty much DOA. But at the same time “service oriented architectures” (Lower Case) were very much alive and well on campuses. We are using plenty of “services” – we just don’t see them as such.

Continue reading →

Update: Comments at the bottom of this post
I must admit that I was 100% surprised when I heard the news that Blackboard had acquired Angel Learning. I would have never even guessed that this would be Blackboard’s next move.
But then a moment later, the absolute brilliance of the idea overwhelmed the surprise. Angel Learning has strong customer growth when Blackboard is doing well in revenue but not gaining customers. Angel Learning has made some inroads into K12 – this is a hard market to crack and the first mover will have real advantages – particularly as sales folks make relationships.
My very next reaction was one of being a little bummed out about Sakai. I really wished that we had invested a bit more energy into improving the 2.x branch to give the market a stronger alternative (Sakai 3 is still a while away). While Sakai 2.x is a great LMS – Sakai historically has had trouble winning straight-up competitive bid situations. Most new Sakai adoptions are because of second-order reasons (all good) like community, controlling one’s own destiny, and seeing the importance of non-course use of a collaborative system like Sakai. All these are the reasons I am so happy my school runs Sakai – but they just don’t win those RFQs at LMS renewal time.
My next reaction was, “What about standards?”. Angel Learning has been a great supporter of standards efforts over the past few years. They shipped IMS Common Cartridge before the spec was completed – that is awesome commitment to supporting standards. Market #1 companies like Backboard naturally lag in adopting standards and Market #2 and #3 companies push for standards like crazy. So when the #2 is merged into the #1 – you expect to lose some momentum.
We shall see if this means that folks from Angel stop showing up to IMS meetings and Blackboard just sends one rep for all three of their products (Blackboard, Vista, and Angel). This is not an impossible situation – it just means that if this results in net fewer active participants – others will need to step up to the plate.
Next I started thinking about the Angel customers – I heard some talk about some spirited discussions in the Angel Learning lists and in the Educause forums. This is natural – some fraction of Angel customers chose “Anything but Blackboard” and now they feel like they have been “re-captured”. This is sad to me – I hope we pick our LMS based on what is best for our teachers and students – not for political reasons – and in particular I really dislike positions which are negatively formulated – in a sense they are inherently unjustifiable.
So what might be the real negative impact on current Angel Learning customers? I think that the most appealing aspect of Angel Learning is that they are a “porous” organization. They let large and responsible customers look at the source code and make changes to the source code. Angel Learning makes patches quickly and gives them to advanced customers very easily. When I talk to folks at large midwestern universities who have Angel – they pretty much tell me that “open source” is not a big deal for them (i.e. they don’t want to switch to Sakai) because with Angel – they have as much of the source as they need – and are happy not to be responsible for the “whole thing”. In a sense, these customers have the best of both worlds – particularly if the school is not interested in being a real developer in a worldwide community like Sakai which takes a lot of energy and commitment.
One of the great complaints about Blackboard is how they lock things down. The reason cited by Blackboard is one of tech support – Blackboard does not want customers breaking things and then blaming Blackboard – a reasonable statement on its face – but for really large sites – some flex is needed. At large sites, a software vendor often is not the expert in their own software as it is running at those sites. Working collaboratively is a great thing. Perhaps the real underlying issue is that Angel is better designed internally to be altered a bit here and there without breaking – Angel evolved in this environment where the vendor and customer work more collaboratively to insure success. For the Angel transition to be successful, Blackboard will have to resist changing Angel to use the “Blackboard way” when it comes to these strategic customer relationships.
The consequences of a mistake on Blackboard’s part would be pretty grave – generally these acquisitions work because very few (usually) small customers get mad and switch just to switch for switching sake. But usually large customers grumble for a while and then simply adapt to the new rules and the new landlord. But if Blackboard changes the nature of the relationship with large customers in a significant way – the large customers might start to switch – and if those schools start to switch – I bet they will switch to Sakai – and this will be a large influx of new developer talent for Sakai – and those developers will add Angel-like features to Sakai – making Sakai a better product. So – word to Blackboard – listen to these folks and keep these folks happy – because if you don’t, Sakai is always waiting.
For mid-tier customers who are going to switch for switching sake – or switching just to be “anything but Blackboard” – be careful how you think about this. First, don’t upend your faculty’s lives just because the IT department is steamed – I am a faculty member and I am telling you that this pisses us faculty members off. LMS transitions should not happen on a whim and they should not happen in a rush. Also, if you switch to some other for-profit company and that company experiences a ton of growth in their customer base – your new company might be purchased in a few years by Blackboard. If you set your LMS strategy based on who the shareholders of your software happen to be – you are making a mistake.
This does make an interesting point about the value of getting software from non-profit companies. Companies like Etudes(*) and CampusEAI are non-profit – if they gain a lot of customers – they cannot be purchased – because the “owners” of a non-profit do not get the proceeds. Sakai Foundation is a non-profit for this very reason – the Sakai Foundation holds the copyright on Sakai – but since it is non-profit we can trust that the Sakai Foundation won’t sell Sakai when it received some attractive offer.
While it sounds preposterous on the face of it – both rSmart and MoodleRooms *could* be purchased by Blackboard sometime in the future because they are for-profit and “have customers”. Often the goal in a take over is to acquire customers locked into contracts as much as it is to acquire intellectual property. So when you purchase open source software from a for-profit company you need to calculate your exit strategy. rSmart nicely provides you with the full source code of their software which you can download – so you could keep using the software and hire another company to give you support on rSmart’s Sakai distribution. This is a reasonable exit strategy – you might want to snag a copy of each of the rSmart source releases and keep it handy in case you need it someday. Just the fact that rSmart gives you this option makes rSmart less attractive (valuable) as a takeover target – and in my opinion the policy of releasing all source is good for rSmart’s customers since the rSmart version of Sakai is not the same as the Sakai Foundation version of Sakai.
I cannot comment on the extent to which MoodleRooms gives its customers an exit strategy because I simply don’t know much about MoodleRooms but it is something for MoodleRooms customers to contemplate. But even though Moodle itself is open source – MoodleRooms may not release *every* line of code necessary to replicate the MoodleRooms environment on your servers. Any non-open source magic glue bits make it harder for customers to switch vendors – and make those companies both more valuable and more susceptible to takeover. (I am not suggesting this is the case for MoodleRooms – just something to think about).
So what do I think we should all do about this acquisition:
(1) Be patient – my opinion is that Blackboard has learned a lot from its WebCT acquisition. If Blackboard avoids big mistakes and takes its time with any changes, this could end up better for customers.
(2) Keep an eye on Blackboard’s “opening up” initiatives – make sure that Blackboard continues to participate in standards and continues to encourages portable approach to software (IMS Learning Tools Interoperability) and portable content (IMS Common Cartridge). When these specs are ready – Blackboard needs to put these into their product and support them wholeheartedly. Since the specs are not yet 100% complete – this is still a bit of a wait and see. The market needs to hold Blackboard’s feet to the fire and also celebrate when Blackboard moves in the right direction. Behavior is best changed by a combination of positive and negative reinforcement.
(3) Support your local non-profit – if you have not done so everyone should join the Sakai Foundation and IMS *right now*. The market needs vital non-profit organizations to keep the market “fair” – think about how bad it would be right now if this happened and there were no Sakai and no IMS. For those who are not members – other folks have been footing the bill for your “insurance policy” / “safety net”. I am very disappointed in the level of University participation in IMS – this should change – we need the customer voice in standards – I know the meetings are tedious at times – but this is about supporting organizations that balance the market.
(4) Immediately install Sakai in addition to your current LMS – fully integrate it – add a logo and a skin and plug in your course roster data – let folks play with it – let adventurous faculty teach with it – use it for project sites for students and faculty – don’t train folks on it – let them learn on their own – give limited tech suport – Sakai was designed to be picked up without formal faculty training. If folks like it, usage will grow – if your users don’t like it, all you lost was a few servers and some disk for a few years. And if you get peeved at your for-profit vendor at some point in the future – viola – you are already several years into your LMS transition! Or perhaps we will get IMS LTI 2.0 really working and your faculty can teach using any tool from any LMS they like and we can combine Blackboard, Angel, Vista and Sakai tools together in one learning portal. Whatever the future holds – getting a Sakai up and running on your campus is a *good thing* and gives you options and amortizes transition costs.
All in all whenever something like this happens – we all need to look forward and find the opportunities that this makes possible. And in all we do and how we approach everything – remember that this is about teaching and learning and not about IT.
Comments welcome – send them to me via E-Mail and I will add them below. I am happy and eager to correct any factual errors herein that you bring to my attention.
(*) Disclaimer: I am a board member of the non-profit Etudes corporation (www.etudes.org)
Comment from Ian Dolphin
Re the above paragraph: