Author Archives: Charles Severance

Tsugi Support for WebSockets

I just finished adding support for a WebSocket based notification service to Tsugi.   It is not a full-up WebSocket server or SocketIO server where you can put a bunch of business logic and data storage in the socket server – it is just a low-latency notification service that can be used to make applications quicker and more efficient compared to techniques like polling.  Here is some documentation on the approach:

http://do1.dr-chuck.com/tsugi/phpdoc/Tsugi/Core/WebSocket.html

Here is a simple test harness that exercises Web Sockets that you can run in your Tsugi:

https://github.com/tsugiproject/socket-test

Here is a more complete chat application with messages, presence, and history and uses notification to work better.  It functions whether or not WebSocket notification is available – it is just more sluggish and has a performance impact when WebSockets are not available:

https://github.com/tsugitools/michat

You can play with the Chat tool at:

https://www.tsugicloud.org/tsugi/store/details/michat

I am going to record a walkthrough video of socket-test and michat sometime soon.

Experiences with Amazon Aurora Serverless

Recently I moved all my Amazon-hosted servers for my www.tsugi.org project from Amazon server-based Aurora to Amazon Serverless Aurora.  I was experiencing the classic problem where 98% of the time it vastly over paying for my servers and then when I hit a peak (like after I spoke at the Instructure conference), things would get slow.  Then I would upgrade my hardware and pay even more when things were slower.  So serverless is *perfect* for me.

I made a serverless cluster – in Aurora serverless you just make a cluster and do not make instances.  I was looking a this Amazon page – https://aws.amazon.com/rds/aurora/serverless/ and it has said:

You pay on a per-second basis for the database capacity you use when the database is active, and migrate between standard and serverless configurations with a few clicks in the AWS Management Console.

I could not find the “migrate with a few clicks” bit in the Amazon user interface.  So I asked a question on Stack Overflow .

When I did not get an answer in a few days, I did the conversion two ways with different results so I figured I would share my results here. I would still love to hear a better approach.

  1. When I did the conversion using mysqldump and restore, with a short outage things were fine.
  2. When I used AWS Database Migration Service it went pretty badly.

First, you have to get the binary log format as “ROW” and binlog retention to 24 hours. That necessitated server restarts on my old clusters.

Then when I got the data migration working, the new database was badly broken.  I lost all my auto increments, then NULL settings in my columns, the UNIQUE clauses and foreign keys in the new tables.

Literally the only thing that migrated correctly was that the actual data and PRIMARY KEY indications. Also, I would recommend migrating one database at a time (i.e. schema) and don’t try to migrate the mysql internal schemas. I said “migrate everything” and the migration tool tried to migrate the MySQL stuff – sheesh – you would think it would know better.

The one thing the AWS Database Migration Service did that was really cool was the migrate and monitor (made possible by the binary logging on the rows). You could watch it moving rows.

So the lesson is use mysql dump and restore  when moving from Aurors server based to serverless.

Sakai 12.2 Maintenance Release Is Available

This post is taken from a Wilma Hodges note to the Sakai mailing lists.

Dear Community,

I’m pleased to announce on behalf of the worldwide community that Sakai 12.2 is released and available for downloading [1] !

Sakai 12.2 has 105 improvements [2] in place including

  • 21 fixes in Assignments
  • 18 fixes in Tests & Quizzes (Samigo)
  • 8 fixes in Gradebook
  • 6 fixes in Forums
  • 6 fixes in Portal

Other areas improved include:

  • Chat Room
  • Lessons
  • Membership
  • Messages
  • PostEm
  • Preferences
  • Profile
  • Resources
  • Site Info
  • Statistics
  • Worksite Setup

There were 2 security issues fixed in 12.2 (details will be sent to the Sakai Security Announcements list).

Please also note the upgrade information page [3] for important notes related to performing the upgrade. 2 Quartz jobs need to be run to complete the conversion steps for Sakai 12, including a new one for the Job Scheduler in 12.1.

[1] Downloading information available at – http://source.sakaiproject.org/release/12.2/

[2] 12.2 Fixes by Tool –  https://confluence.sakaiproject.org/display/DOC/Sakai+12.2+-+Fixes+by+Tool

[3] https://confluence.sakaiproject.org/display/DOC/Sakai+12+upgrade+information

Deprecate and Remove support for IMS LTI 2.0 from Sakai

At this point, all the major LMS vendors that support LTI 2.0 have quietly decided to move away from LTI 2.0 and towards LTI 1.3 and ContentItem and follow-ons.

LTI 2.0 has a few good features and a lot of bad ideas.  Sakai’s LTI 2.0 (and almost LTI 2.1) is one of the best in the industry and one of only two formally certified LTI 2.0 LMS’s.

But even with all those positives, LTI 2.0 in Sakai is many lines of intricate code that if never used is a liability for security and maintenance – and as new features like LTI 1.3, its services, and new follow-on specs emerge – it is nice to clean up a large amount of code to make space for new stuff.

So at this point LTI 2.0 is deprecated in Sakai 12.x and will be removed in an upcoming release of Sakai.

If you have any concerns – or tools that require LTI 2.0 – please note them in this JIRA:

https://jira.sakaiproject.org/browse/SAK-40065

Or let me know using some other channel.

Sakai 12.1 Released

The text of this post is taken from the announcement made by Wilma Hodges – The Sakai Community Facilitator.

I’m pleased to announce on behalf of the worldwide community that Sakai 12.1 is released and available for downloading! [1]

Sakai 12.1 has 97 improvements [2] in place including

  • 20 fixes in Gradebook (aka GradebookNG)
  • 9 fixes in Tests & Quizzes (Samigo)
  • 7 fixes in Assignments
  • 6 fixes in Resources
  • 5 fixes in Lessons

Other areas improved include:

  • Calendar
  • CKeditor
  • Dropbox
  • Forums
  • Portal
  • Preferences
  • Profile
  • Roster
  • Section Info
  • Security
  • Signup
  • Site Info
  • Statistics
  • Syllabus
  • Web Content

Four security issues fixed in 12.1 (details will be sent to the Sakai Security Announcements list)

Please also note the upgrade information page [3] for important notes related to performing the upgrade. 2 Quartz jobs need to be run to complete the conversion steps for Sakai 12, including a new one for the Job Scheduler in 12.1.

[1] Downloading information available at – http://source.sakaiproject.org/release/12.1/

[2] 12.1 Fixes by Tool –  https://confluence.sakaiproject.org/display/DOC/12.1+Fixes+by+Tool

[3] https://confluence.sakaiproject.org/display/DOC/Sakai+12+upgrade+information

 

Sakai Community Leadership Update

By now, you know that Neal Caidin will no longer be our full-time Community Coordinator as of April 30, 2018 because of a reduction in Apereo membership revenue for Sakai. We still have funding – just not enough for a full-time coordinator. We will miss Neal’s smile, talent, and tireless dedication to our community.

http://www.dr-chuck.com/csev-blog/2018/04/sakai-coordinator-funding-in-2018/

The Sakai PMC, Sakai Commercial Affiliates, and Apereo Board have been coordinating on how to fill the roles that will be left open when Neal leaves. I attach a draft of a document we have been developing that looks at the community leadership roles in general that are currently filled by Neal and others.

We will fill these roles by a combination of volunteer leadership and by paying for time. Some of these roles are 20 hours per week and so we will use Sakai membership money to fill them. Here is a list of the people who will be leading us in these roles going forward; where we have a lead / back-up in the position, there are two names:

  • Community Facilitator: Wilma Hodges (Longsight)
  • Release Manager: Matthew Jones (Longsight)
  • Branch Manager: Matthew Jones (Longsight) / Earle Neitzel (Longsight)
  • QA Lead: Dede Hourican (Marist) / Jolie Tingen (Duke)
  • A11Y Lead: Tiffany Stull (Virginia) / Terry Golightly (Johnson)
  • I18N Lead: Miguel Pellicer (EDF)
  • Security Lead: Miguel Pellicer (EDF)

At this point, we have worked out an arrangement with Longsight to cover some of Wilma’s and Matt’s time taking on the Community Facilitator and Release Manager roles. The terms of the arrangements have been shared with the Apereo, the Sakai PMC and all the Sakai Commercial Affiliates. The PMC is making these appointments for 3 months and will review these roles August 1, 2018, and then formally consider others who might be interested in these roles.

The other roles are not directly compensated at this point, so we should thank those who have taken on this leadership as volunteers–as well as others who are not currently listed but are already part of our community leadership. To provide some support to these volunteer leaders, the PMC will retain some of the Sakai member funds and make them available to help with travel and other necessary expenses if a person’s organization is not able to cover them in the individual’s volunteer role. This will not be a lot of money but we feel that it can be very helpful in ensuring that our volunteer leadership makes it to meetings, conferences, and events.

The PMC will monitor the costs of these new approaches and make adjustments over the coming months, including additional compensated roles as finances permit. All funds that are spent will be done so in the open within the PMC and all of the Sakai SCAs. The PMC chair will report a summary of the expenditures around community support to the whole community quarterly.

You are welcome to join the PMC public mailing list to watch this happening more closely if you are interested and want to participate.

https://groups.google.com/a/apereo.org/forum/#!forum/sakai-pmc

Our goals in this staffing activity is to end up with Sakai community leadership that is both broad and deep and can handle turnover in these roles. We want to encourage involvement from all of our SCAs if they have interest.

Release 0.5.0 of Tsugi coming soon

If you have been watching Tsugi-php and tsugi lately, you have seen the pace of commits slowing down.  This is good because ultimately Tsugi wants to be a library that just works.   But a few recent efforts like improving blob support and activity tracking took a lot of effort and a lot of commits. But that code is now solid and has been in production for a while at scale.
The Tsugi release approach is that you run master and in a periodic cron (for me it is hourly) do a “git pull” and run upgrade.php.   Master needs to always be production ready and all my production servers run master within an hour of a commit.  While many of you may not be this aggressive (and don’t need to be) – I do this so I know if something goes wrong almost instantly.   We are working on better unit tests and post-commit hook functionality tests to make this strategy even safer.
But with this as approach, the way releases work is more to do with freezing master at a very safe place so that we can try something in master.  That way if something goes awry in master – you just jump back to  a safe place.  Kind of like mountain climbing with a lead climber.  Always have a place to fall back to.
I will be making a change in master that I expect will be safe, but just to be sure, I want to make the 0.5.0 release / branch to come back to.
Some of you have noticed that I have frozen various versions of dependencies in composer.json:
I did this because I did not want to update a dependency and mix it with a change I was making as we went along the path.  This way I did not make my change and update a dependency in the same breath so I *know* if something broken is my fault or a dependency failure / regression.   Dependencies are software too – and can make make mistakes in minor releases.  I conside my own QA to be QAing both all of tsugi plus its dependencies.
So what I will do as soon as I make the 0.5.0 release is update composer.json to let the dependency versions upgrade.  I will do this in a series of steps to allow for some QA/production experience at each step.
I will try to make no changes in tsugi-php or tsugi during this time unless a bug appears or a dependency upgrade breaks something.
What you will see is a series of updates that update a *lot* of files – all in the vendor folder.   It is OK.

Sakai Coordinator: Funding in 2018

The Apereo Foundation is five years old. In that time we’ve grown from a handful of software communities serving higher education to almost twenty. The network of contributors to our software communities has grown significantly, encouraged by the global partnerships we forge.

We have structured our membership of educational institutions and commercial affiliates to be as inclusive as possible, and to reflect differing levels of financial resources available to higher education internationally. Large institutions are and remain important – but the voices of smaller institutions, or those with less available resource, should not be excluded or lost.

Membership subscriptions and in-kind contributions support everything we do. We have no endowment or other sources of revenue. Our membership structure has two main components – a core membership subscription, which supports the functions and activities the Foundation provides for all our software communities, and individual software community subscriptions supporting the specific activities of our constituent software communities.

Last year Foundation membership grew by more than twice the numbers we lost. In overall financial terms, our core membership revenue more or less broke even. Sakai supporting subscription revenue did not meet its target. In these circumstances, reluctantly, the post of Sakai Community Coordinator is being terminated effective April 30th.

Neal Caidin, the current postholder, has done an amazing job, and this decision does not reflect on the ability, enthusiasm and good humor Neal has brought to the role. In addition to Neal’s management of the Sakai release process worldwide, Neal has been a valuable coordinator and contributor from the introduction of the Lessons tool in 2.9 through the responsive and mobile design of Sakai 12 to the WCAG 2.00 AA certification effort – and much more. Sakai has matured to become a more inclusive and effective community as a result of Neal’s leadership and capacity to facilitate with sensitivity and empathy. Apereo itself has benefited from Neal piloting processes such as Funding and Resource Management (FARM), feature prioritization, and conference planning. We would like to thank Neal and to wish him well on his next ventures. Neal’s last day is April 30.

Sakai is losing a Community Coordinator, but is not losing community coordination. In the coming weeks the Sakai PMC will work with the Foundation to evaluate the best means to deploy available resources to meet the needs of the Sakai community. The Sakai Supporting Subscription will continue to make a very significant contribution to meeting those needs. We will conduct as much of this work in the open as possible, and ensure regular community communication and consultation.

Chuck Severance, Chair of the Sakai PMC
Ian Dolphin, Apereo Foundation Executive Director

Sakai 12.0 is released!

(This blog post is taken from an announcement by Neal Caidin the Sakai Community Coordinator)

Congratulations amazing worldwide community!

Our community wiki has both functional [1] and technical [2] release notes.

Download instructions are also available of course [3] .

Several of the top changes include:

  • Improved look and feel. New skin. More attractive and functional.
  • Improved responsive design, better performance on mobile devices.
  • Gradebook – performance enhancements, plus a number of minor interface improvements such as progressive rendering of student grades for large classes, and a new feature which allows instructors to view and adjust final grade distribution.
  • Assignments – Refactoring of Assignments tool. Makes Assignments more maintainable and easier to enhance.
  • Lessons has a new optional subpage left menu navigation feature. In addition, a number of useful enhancements such as a forums widget, announcements widget, calendar widget, resources widget, and personalization of text.
  • Test and Quizzes (Samigo) – Extended delivery of assessments for select individuals and groups.
  • Chat tool – updated interface.
  • Pluggable ContentReview which contains 4 plagiarism detection integrations.
  • Standards – support for IMS Content Item, including a “shopping cart” ubiquitously available in the rich text editor.
  • Accessibility improvements.

[1] Functional release notes – https://confluence.sakaiproject.org/display/DOC/Sakai+12+Release+Notes

[2] Technical Release notes – https://confluence.sakaiproject.org/display/DOC/Sakai+12+Technical+Release+Notes

[3] Download instructions – http://source.sakaiproject.org/release/12.0/

 

Abstract: Building Learning Tools for Canvas using the Tsugi Application Environment

This presentation will introduce users to the free and open source Learning Application development environment available at www.tsugi.org. Canvas supports the IMS Learning Tools Interoperability to allow external tools to be easily integrated into Canvas and used to increase the richness and diversity of the course experience.   External tools can provide functionality not present in Canvas like a course map tool, or social grading system, or a YouTube viewer with Caliper analytics.   External tools can also be rich auto-graders that make online classes bother more interesting and more effective.  Imagine if you could easily develop an interactive system like Pearson’s MyMathLab for *your* course and topic.  Tsugi provides a set of libraries that allow you to write learning applications without any need to read an LMS integration specification.  Tsugi’s run-time environment takes care of complying with the necessary standards to make plugging Tsugi tools into Canvas, or any other LMS.  Tsugi also provides a hosting environment that is as easy to deploy as WordPress and allows institutions to host tools and provide an “App Store” of available tools to their users.  There is also a free cloud-based Tsugi App Store that provides scalable hosting for open source Tsugi applications at www.tsugicloud.org.  It is easy to learn to write Tsugi applications using the online materials at www.tsugi.org.