Monthly Archives: March 2021

Sakai 21.0 Released!

This is from our wonderful community coordinator – Wilma Hodges.  I just pasted it into a blog post for the record.

I’m pleased to announce that Sakai 21.0 is now available!

Many thanks to everyone involved with the release! It wouldn’t happen without all of you!

Our community wiki has both functional [1] and technical [2] release notes.

Download instructions are also available [3].

New in Sakai 21

  • Dark Mode
  • Dashboard – New tool!
  • Lessons improvements, such as revamped Add/Edit Dialog, new Add Layout template options, improved Reorder and Date Release Indicators, and more color and formatting options for headings, buttons, and borders.
  • LTI Advantage Improvements, including autoconfigure option and new LTI Assignment Type in the Assignments tool.
  • Gradebook enhancements for exporting category averages and messaging students from the gradebook
  • Rubrics are now searchable and can contain weighted criteria
    And more! [4]

[1] Functional release notes –

[2] Technical Release notes –

[3] Download instructions –

[4] Complete Feature Summary –

Wilma Hodges, Ed.D.
Sakai PMC – Community Coordinator

Open Is As Open Does

The recent announcement of the OpenLMS making its extensions to Moodle available as Open Source generated a bit of a dust up in the EdTech journalism space.  Here are several great articles:

  • Phil Hill wrote a piece titled “Moodle’s Dispute with LTG and its Growing Suite of Former Moodle Partners” that focused on the relationship between Moodle and OpenLMS.  OpenLMS keeps buying Moodle commercial partners as soon as they get a decent number of customers and switching the customers from the Moodle community edition to the OpenLMS edition.  It highlights the difficulty of making a 100% open LMS sustainable (i.e. getting sufficient funding to support a team of 50 or so).
  • Michael Feldstein wrote a piece titled, “Moodle’s Sanctimony on Openness is Moot“.  It is a fun read as it wanders around the well-known fact that virtually all open source products are available from commercial vendors and those vendors add a bit of glue so the products run cheaply and reliably at scale in hosting environments like Amazon.  Michael takes the somewhat narrow position that if a little bit of code is not open then the whole “claim to be open” kind of falls apart (greatly simplified – ed.) – if there is a single crack in the “perfect open” – then there is no point in loudly arguing subtle differences.

Please read these before reading my post – as my post is very much additive to the conversation and I will not repeat what both of the above posts have accurately and elegantly said about the situation.

Also before reading further, understand that this is not me grinding an axe on my enemies or competitors.  I am friends with Martin Dougiamas (Moodle),  Phil Miller (Open LMS),  a number of Canvas employees like Karl and Melissa as well as friends of Phil Hill and Michael Feldstein.   I cherish those friendships so this post is not to bash anyone – it is just looking at the current situation through my personal lens.

Open is as Open Does

I think that the major flaw in both of the above pieces is that they look at the situation through very literal non-gray-area lenses like.  “What is the license?” or “Does Martin have a legal right to be upset about OpenLMS?” or “Where is that hidden code?” or “Is there a moral right or wrong when something is claimed to be open?”  These excellent “in depth” pieces are flawed because they are “in depth” and as such miss the bigger picture and don’t explore the all important gray area that matters most.

The bigger picture is that the purpose of “open” is to empower others to fully participate in a software ecosystem and contribute to its directions without requiring a commercial relationship.

I will look at the major “LMS” projects that will provide a Git repository with an OSI-certified license – that is the pure definition of “de-jure open”.    But that is only a small part of the “de-facto” definition of open.  I see a number of factors that are very important to non-paying adopters where I will grade each LMS in the grade book below:

  • How many non-paying servers have been installed?  This is where the rubber meets the road and the analysis could almost stop here.  Ultimately the other factors in this grade book lead to better or worse adoptability and that drives installation count.
  • Is there a structured community where non-paying participants have equal standing as paying participants.  For example, are there conferences?  Are these “paying customers only” or are they open?  Are the conferences about building community or herding customers into rooms with free drinks and snacks once per year.  Are there active developer lists, IRC’s or Slacks where folks can and do participate?
  • Is there a real way for non-paying adopters to get support for their product?  If they install a new version and see a weird message during database conversion, where do they go? Who will respond?  If they find a bug, can they submit a patch?  If they submit a patch will it ever make it into the core source code and a released version of the product?
  • Can non-paying adopters be part of the core team of developers?  Can they be part of the core decision making process?  Do they have access to the leadership team and can they make their case for a feature to the actual person who will build the feature?

All of these are variations on an organizational value of “welcoming, celebrating, and appreciating non-paying adopters”.

Since I am a teacher, I gave four LMS’s with some aspect of “open” in their nature grades in these categories.

The Grade Book

Non-paying participants
LMS License Server Count Community Support Leadership Access
Moodle GPLv3 A+++ (> 100K) A A+ B-
Sakai Apacheish B (> 100) A A A++
Canvas AGPL D (Some) D F F

Comments on the grades

  • Moodle is the best / “most de-facto open” in this analysis.  They “do open” better than anyone else.  They are three orders of magnitude better than all of the others combined in the most important metric of non-paying servers installed.  Their community is wonderful – their idea of teacher-centered MOOTS is *brilliant*.  If you have a problem with Moodle you post to the list and will likely get an answer from almost any time zone in the world and in almost any language.  #Jealous The only knock I have on Moodle is the insular nature of their core team.  They are treated as celebrities at Moodle events but then run back and hide in the walled garden of HQ and make all the decisions.  Fixing a bug or making a small contribution to Moodle requires you to become friends with folks on the inside and leave your offering at the door and hope someone comes out and picks it up.
  • Sakai (my entry in the list) is much smaller than Moodle in terms of servers installed.  Sakai’s events and other community activities are an even split between teacher and developer focus.  We have many examples of large and small contributions from relatively unknown non-paying community members.  If the code is good and follows the style guide and they fill out the contribution agreement – it goes in.  It is sometimes a little challenging to know the right way to do something in a nearly 20 year old code base with 1.4M lines of code. Sakai has a 1.5 hour online core meeting *every week* with open access that anyone can attend where our most core, senior, and talented engineers, UI experts, and accessibility experts will review github PR requests from anyone on the planet and help the person re-work their changes so we can get them in.  Sakai’s strongest column is “Leadership Access” – you can monitor core meetings by just joining online – Sakai has over eight meetings per week that are open to the public.  These are the only meetings where things get done and decisions get made.  “Sakai is 100% open – all the way to the core”.
  • Everything about Canvas derives from the AGPL license and “Open Core” approach.  I won’t nick them on Open Core because frankly the core is enough of a decent LMS for a smaller organization to use that the non-core bits are not all that important.  And the bits to host at scale are not very reusable anyways.  Canvas does not encourage non-paying adopters or small commercial companies hosting Canvas – but it happens.   I have no idea the extent or the number of schools and companies that run the free Canvas.  I think this is more prevalent in places like China and India where (a) Canvas does not have much presence, and (b) the countries have a wealth of technical talent and can handle a Ruby-On-Rails application without asking a developer list for help.  Canvas does a good job of maintaining the instructions on how to install the open version but does not really welcome non-paying adopters into “the family” with open arms.  Also the core Canvas staff (who are awesome by the way) are super insulated from any customers (paying or otherwise).   Their “vote on new features” web site is a laughingstock – but consistent with a core value that non-paying adopters are only useful as a marketing gimmick to assure the infinitely more important paying customers.  But kudos to Canvas – their minimalist approach to openness has made a decent LMS available to countries and communities that can make use of the product.
  • And now we come to OpenLMS.  To be fair, OpenLMS has only been open source for a month.  Almost all their grades are TBD.  There is a nascent community at – there is an Open LMS Open Roadmap: What To Expect blog post you should read.  That is all we get.


Since OpenLMS is only a few weeks into their Open Source experiment, OpenLMS / LTG is mostly “potential” and they will be making some decisions along the way.   They can look at this blog post (and its grades) as one possible roadmap to “open greatness”.  I think back to the “score card” that Ray Henderson did at a few Blackboard meetings – OpenLMS could use my grades and categories as their starting point and goals.

OpenLMS could track and regularly publish how many non-paying adopters use their product.  They could have a series of online forums and Slack channels staffed by OpenLMS staff to help those not paying any money for the product with online support.  They could even open source some of their “scaling magic”.  They could have their core design and developer meetings open to the public.  They could be the most open open source LMS ever!

But… (there is always a but)  LTG is a public for-profit UK company.  Companies are not known for their largess.  Phil Miller is a great friend and really cares about his paying customers.  He loves to build clever software to keep his development and hosting costs low (some of the best in the industry by the way) and pass some of the savings on to his customers.  And it is why LTG loves (and should love) Phil Miller.  I have learned a lot about how to build inexpensive scalable systems (like Tsugi) from having beers with Phil and his teams over the years.

My opinion is that LTG/OpenLMS will evolve to follow the Canvas path – in essence the high level goal is compete with Canvas head-to-head.  Release it open source – don’t worry too much if no-one bites – and if there are some markets that OpenLMS is not interested – some tiny companies can use OpenLMS and sell services to a few local schools.  But with Moodle available and a known commodity it would be hard to find that much extra value in  OpenLMS.  If I were a small school and interested in OpenLMS after I downloaded it and ran it on my desktop – I would probably just pay LTG for hosting and let them deal with all the complexity.  Oh wait,  LTG just acquired a new paying customer who was attracted to the free version as bait.

Only time will tell.  But at least this blog post will be available in the Internet Archive and we can all come back and see if my predictions were accurate :)

P.S. Oh yeah – in EdTech I far prefer to deal with non-US companies and for those companies to be public – not private.   I am quite unhappy about the LMS market trend towards large privately held US companies.  So I like the fact that LTG is UK and public.

Should Sakai Evolve Towards MySQL 8.0 or Maria DB 10?

With the recent release of Sakai 21.0 (Thanks to all involved). We need to do some thinking about our direction forward w.r.t. how we approach our MySQL support in the Sakai community.

We have been supporting MySQL and Oracle since 2002 (19 years). In the old days, MySQL was the 100% open alternative for open source applications. But then in 2010, Oracle gained control of MySQL by buying Sun Micro-systems. I wrote a pretty fierce blog post about the Oracle purchase of Sun and my concerns for the future. At that point in time MariaDB was forked from the last non-Oracle version of MySQL. Some in the Sakai Community experimented with MariaDB from time to time – in the early 2010’s if things went wonky we ran back to MySQL.

Oracle has released MySQL 5.6 and 5.7 and MariaDB kept pace – even to the point of binary interoperability. We along with the rest of the open source world found this equivalence comforting and felt it was OK to just keep using MySQL and let the “other folks” use MariaDB.

But support for MySQL 5.6 just expired (February 2021) and support for MySQL 5.7 expires in October 2023.

The version of MySQL after 5.7 is MySQL 8.0. This is the first version that is heavily influenced by Oracle and it feels to me like MySQL 8 and MariaDB may begin to diverge. Hibernate (as of 5.3) supports the org.hibernate.dialect.MariaDBDialect separately from its MySQL dialects. Hibernate also supports the org.hibernate.dialect.MySQL8Dialect.

The reasonably high likelyhood is that Sakai will be able to simultaneously support the various versions of MySQL and MariaDB without too much effort. About 25% of the SQL in Sakai is hand-constructed and 75% of the SQL is constructed by Hibernate. Our hand-constructed SQL does not make use of intricate features of MySQL so it is likely to keep working in MySQL 8 and MariaDB 10. And you can pick your Hibernate dialect and support all the databases.

So the question is less about *not* supporting MySQL 8 that it is about deciding what our “preferred” / “first choice” database will be. For this preferred MySQL compatible database we can encourage developers to test using the preferred database and then run our nightly servers using that preferred database.

For example for Sakai 21 on our nightly page, we have eight MySQL 5.7 servers and one Oracle server.

For Sakai 22, we could keep things the way they are or switch to any the following:

* Run eight MariaDB 10 servers, 1 MySQL 5.7 server, and 1 Oracle Server
* Run eight MySQL 5.7 servers, 1 MariaDB server, 1 MySQL 8 server, and 1 Oracle Server
* Run eight MariaDB 10 servers, 1 MySQL 5.7 server, 1 MySQL 8 server, and 1 Oracle Server
* Run eight MySQL 8 servers, 1 MariaDB server, 1 MySQL 5.7 server, and 1 Oracle Server

You get the picture. By leaning towards either MySQL 8 or MariaDB 10 – we are signaling our “preference” / “first choice”.

We of course would fix problems that emerged in any of those databases. We would not want to break Sakai and MySQL 8 just to do something cool in MariaDB 10 or vice versa. We need to write conservative SQL and accept fixes if some hand-written MySQL breaks 5.7, 8.0 or MariaDB 10.

The question is where the majority of us are going to go so we can stick together and protect each other’s flanks by using a common approach wherever practical.

We can delay this decision until we release Sakai 22 – but the decision is easier and safer now. We get a year to experiment with the different databases and then based on what we learn this year – we can make a more informed decision next year. Also by next year we might get some signals from the market and other open source projects as to where they will be going w.r.t. the “MySQL 8 – to be or not to be” question.

Once we chat a bit about this on the lists, I will do a survey to poll the community.

My Opinion

At a minimum, I do *not* want to go “all in” with MySQL 8 – I want to move toward MariaDB if a year of experiments reveals no issues.

I would like a trunk master running on MySQL 5.7, MySQL 8.0, MariaDB 10, and Oracle. I will run my Smoker process against all four every night, to probe for really bad regressions. If we could somehow make them have identical sites and data – I could compare them click for click with smoker.

Earle Nietzel’s Opinion (from email)

I do think it’s time to start more formally adding MariaDB to the mix, however I think our database setup on nightly should reflect that which is being used by our users meaning:

For example:
80% of Sakai installs use MySQL (where 20% of installs make up 10% Oracle and 10% MariaDB – hypothetical numbers). For the other DB variants that make up the minority lets simply have a version of master representing that DB.

So an infrastructure might look like this:

master – 1 mysql / 1 mariadb / 1 oracle
experimental master – 1 mysql
21.x – 1 mysql
20.x -1 mysql
19.x – 1 mysql

Also the SQL ratio according to tables is: ~300 total tables of which 75% are Hibernate managed and the other 25% represent SqlService and Spring jdbc template.

Another thing to be thinking about with all these instances of Sakai is whether or not we have the resources to test on them.

I am not in favor of expanding instances if we don’t have resources to even test on them. This is the reason I suggest having only master instances for all of the databases we support and other instances are the db that is used most by users of Sakai.

Matthew Jones’ Opinion (from email)

I don’t know if that accurately represents the community or not. I feel many use AWS Aurora, another fork that is not compatible with MySQL 8, and only goes as far as 5.7, similar to MariaDB. So do we need an Aurora instance too? Probably not as much as if we had MariaDB.

I like the idea of going MariaDB first as it’s the most open, 5.7 compatible and it’s also the easiest for localhost. Though MySQL 8 isn’t too big of a problem to continue supporting as long as we don’t do much MySQL 8 specific stuff, which we likely won’t do. If we’re running MariaDB, I don’t see any point to leaving a separate MySQL 5.7 around, especially since it’s going to be EOL in 2023.

I think the hard decision about supporting multiple databases is if we want to take advantage of any special features of a database, we can’t while still supporting the others. For instance MariaDB supports a number of storage engines that aren’t available on the other databases and features like system versioned tables, that could be “nice to have” if we were able to use them. We’re basically stuck forever at the lowest common denominator, which is going to be MySQL 5.7 in this case.

Is OER always free? Always?

I stirred up a fun hornet’s nest by my answer to this question on the Creative Commons mailing list.

My feeling is that while this is a fun and articulate dust up with writing by smart and thoughtful people.   Nothing will really change w.r.t. funding truly open source and truly open content.  But it is a fun read.

*Chuck’s Answer to “Is OER always free? Always?”*

TL;DR; Chuck is probably too passionate about Open Source and Open Content – feel free to delete this :)

I think that a really key element of this conversation is that even if OER material removes the commercial cost of a license – it does not and cannot eliminate all costs.

You might need to purchase a computer or phone to read it, you might have to need to purchase an Internet data plan, if you want it on paper, you might need to purchase a printer and some paper, or possible pay someone with a printer to print it for you at a reasonable cost.  You might need to rent a server to host it online for those in your courses.  You might need to pay a person to monitor your server and upgrade it from time to time.

The makers of OER cannot eliminate these normal / reasonable costs of access.   These costs have been with us longer than the Creative Commons license has been in existence.  They are not a problem – they are just reality.

*Building to a Rant*

The new cost that is more subtle is when OER content is not useful without some kind of software which may or may not be free.   Perhaps the content is in a non-open source adaptive learning system where you cannot get the software and you cannot even get the raw OER or see the data model of the adaptive learning system so you could build your own.   It is a closed magic box that is opaque.

Because the OER is “openly licensed”, perhaps if you pay to use the access software, you can take screen shots of the OER, then scan it, and OCR it and *legally* make your own copy.  But this is a little weak on the “reuse” claim – you can reuse it at massive cost or effort.  The LibreTexts project puts a lot of effort into “freeing” OER licensed content from proprietary forts.  It is a lot of work – but pretty cool.

And sure you can wave your hands and dream of open content and a complete open source chain of production where the raw material is openly licensed and *everything* in the value chain is also free.  I have done this for *one* course – Python for Everybody.  If you start with my github repo, you can build an LMS, a web site, an online teaching system, and even a camera ready textbook ready for printing using 100% free software.  I use LaTeX and pandoc for the print book – it is perhaps less convenient that Word but it is free.  P.S. I also have a README that tells how to do it.

It is possible with great effort, but there are a number of problems.

First, it is harder to keep a 100% open chain of production from beginning to end – you need solid technical skills and a lot of patience.  I have shown 100’s of people my 100% open process – and literally no one has replicated it because it is easier to just fall into the easier path of proprietary approaches.

Second, no one supports or invests effort into improving the free and open source production chain because (again) everyone is so busy (or lazy).  This help s keep the no-free alternatives easier to use.

Third, philanthropists like Dell, Zuckerberg, and Gates and governments pour money into semi-proprietary activities because those activities feel “comfortable”.  Giving money to open source hippies – not so much.  The short sightedness of these well-intentioned funders makes me see red – they prop up weak companies with overly large staffs using the not-100% open models and never give a single cent to real, open models.

Fourth, efforts that start open source / open content that reach some escape velocity are quickly moved away from the truly open world and towards a partially-commercial, profitable, and sustainable.  As they move away from open, benefit flows towards them in the form of revenue, grants, and accolades (because they spend money on sales and marketing).  They then fork away from their open roots and just become proprietary for some essential parts of their operation to achieve the lock in they need as they go from 5 to 40 employees.

As a 100% open end-to-end person (one of the few) – I get it – and accept that 100% open is always going to be a hard path – it is like being a monk and sleeping on cold stone floors :)

The thing that pisses me off in conversations like these is how those who are “pretty open”  or “open core” or whatever participate in these discussions with the intent of defining their variation of hybrid proprietary + open in the name of money as “good enough” or “the best we can do” or “an ideal compromise”.  They want validation / kudos / accolades for their particular choice of non-open bits.

I get that there are a lot of “not 100% open” business models and those should be “allowed” – but they should not be “celebrated” as the “pinnacle” of open just because someone makes a speech about how *their* hybrid model is the best we can do.

By the way – there is a way to make money on things like printing, hosting, support, etc etc without violating the principle of 100% open.  The Occam’s razor is whether someone else can replicate what you are doing if they put their mind to it – and if your company has software that is essential to making some OER content comes to life.  It helps if you also have a README as to how to set up a copy of what you have built.

It is 100% honorable to make *reasonable* profit for value add on top of open source.   But you make a lot more profit if you sneak some proprietary bit into the value chain and then get folks to adopt your entire chain because has some open elements to it.

Back to your regularly scheduled conversation patting “open core” folks on the back and giving them your money because some open is better than none. :)


P.S. If you want to really confuse me – start talking about ElasticSearch, its revenue model, free versions, and its licensing :)