Monthly Archives: September 2008

Rant: Ill for 1.5 Days because of WSDL, XML, and Schema

Saturday I worked for a while with WSDL, XML, and Schema – trying to make some extremely simple stuff work over SOAP in PHP. I finally got some stuff working but the stress of figuring it out took its toll. I hate being so non-productive – I like getting stuff done – I dislike figuring out crap rats nest data models in XML/WSDL/Schema.
It was so bad that I tried to get the poison out of my system by taking a long motorcycle ride to pick up a new batch of baby Sakaigers – no help. Sunday – I could not work – I spent most of the day laying in bed – mentally exhausted.
Then this morning – I woke up with a splitting headache and knot in my stomach – I could hardly stand up. I took a Motrin and laid back down – after about three hours the gurgling in my stomach ceased and I was able to stand up enough to drive into work for my 3PM meeting.
It is just really sad that WSDL/XML/Schema is so bad that it makes one sick to one’s stomach for a day and a half just by looking at it.
I figured that maybe I should try XMLSpy instead of looking at it all in vi – perhaps looking at xsd files in vi was what gave me vertigo. But of course – XMLSpy is not free – it is $149. Great market – the world has adopted these crappy standards and you have the only piece of technology to make them tolerable – and you only give a trial edition – not a free edition. GRRRRRR.
This made me more sick to my stomach – it has gotten to the point where you pay $149 for the right to view Schema without vomiting. We as an industry have built standards on top of XML and Schema that are so nasty that you need to pay some stupid company $149 to interpret them – where is the free alternative??
So I kept googling “free mac os xml viewer” and came across – Exchanger XML Lite – which was free. Their web site is:
http://www.exchangerxml.com/
Now I can load up the schema files and Exchanger will load the million external references and tell me where I need a string named “Fred” and it is up to 256 characters.
If we want object oriented stuff like extends and want to avoid repetitive code – then lets just make our specs be honestly OO and use open braces and close braces – then at least we can run JavaDoc and see what methods our ultimate object will have. Why add a zillion less thans, colons, and greater thans to completely obscure things.
I need to get back to making SOAP work – but I am nervous because I lost 1.5 days of productivity just by looking at XML schema – so even armed with Exchanger – I am fearful of diving back into the swirling darkness that is XML Schema.
Ah well – maybe writing this blog post will make me feel better. Or maybe someone will tell me a cool way to run a script that auto-generates PHP objects given some crap WSDL as input.
Comments
I’ve been building stuff in XML, XSD, XSL for a couple years using OSP. A good toolkit for free would have been great, but it isn’t out there.
However, OxygenXML has an academic license for $50 and works as an Eclipse plugin. I really like it.
Sean

Google App Engine: Printing out all the Request Parameters

I figured out how to print all the request parameters in Google App Engine:

logging.info(self.request.params.items);

It looks like this:

bound method UnicodeMultiDict.items of UnicodeMultiDict([(u’A’, u’3′), (u’b’, u’4′)])

Along the way, I found useful information by making a dir() of the request object:

logging.info(dir(self.request))

[‘GET’, ‘POST’, ‘ResponseClass’, ‘__class__’, ‘__delattr__’, ‘__dict__’, ‘__doc__’, ‘__getattr__’, ‘__getattribute__’, ‘__hash__’, ‘__init__’, ‘__module__’, ‘__new__’, ‘__reduce__’, ‘__reduce_ex__’, ‘__repr__’, ‘__setattr__’, ‘__str__’, ‘__weakref__’, ‘_body__del’, ‘_body__get’, ‘_body__set’, ‘_body_file__del’, ‘_body_file__get’, ‘_body_file__set’, ‘_cache_control__del’, ‘_cache_control__get’, ‘_cache_control__set’, ‘_environ’, ‘_environ_getter’, ‘_headers’, ‘_headers__get’, ‘_headers__set’, ‘_host__del’, ‘_host__get’, ‘_host__set’, ‘_urlargs__del’, ‘_urlargs__get’, ‘_urlargs__set’, ‘_urlvars__del’, ‘_urlvars__get’, ‘_urlvars__set’, ‘accept’, ‘accept_charset’, ‘accept_encoding’, ‘accept_language’, ‘application_url’, ‘arguments’, ‘blank’, ‘body’, ‘body_file’, ‘cache_control’, ‘call_application’, ‘charset’, ‘content_length’, ‘content_type’, ‘cookies’, ‘copy’, ‘copy_get’, ‘date’, ‘decode_param_names’, ‘environ’, ‘get’, ‘get_all’, ‘get_range’, ‘get_response’, ‘headers’, ‘host’, ‘host_url’, ‘if_match’, ‘if_modified_since’, ‘if_none_match’, ‘if_range’, ‘if_unmodified_since’, ‘is_xhr’, ‘max_forwards’, ‘method’, ‘params’, ‘path’, ‘path_info’, ‘path_info_peek’, ‘path_info_pop’, ‘path_qs’, ‘path_url’, ‘postvars’, ‘pragma’, ‘query’, ‘query_string’, ‘queryvars’, ‘range’, ‘referer’, ‘referrer’, ‘relative_url’, ‘remote_addr’, ‘remote_user’, ‘remove_conditional_headers’, ‘request_body_tempfile_limit’, ‘scheme’, ‘script_name’, ‘server_name’, ‘server_port’, ‘str_GET’, ‘str_POST’, ‘str_cookies’, ‘str_params’, ‘str_postvars’, ‘str_queryvars’, ‘unicode_errors’, ‘uri’, ‘url’, ‘urlargs’, ‘urlvars’, ‘user_agent’]

logging.info(dir(self.request.params))

[‘__cmp__’, ‘__contains__’, ‘__delitem__’, ‘__doc__’, ‘__getitem__’, ‘__init__’, ‘__iter__’, ‘__len__’, ‘__module__’, ‘__repr__’, ‘__setitem__’, ‘_decode_key’, ‘_decode_value’, ‘add’, ‘clear’, ‘copy’, ‘decode_keys’, ‘dict_of_lists’, ‘encoding’, ‘errors’, ‘get’, ‘getall’, ‘getone’, ‘has_key’, ‘items’, ‘iteritems’, ‘iterkeys’, ‘itervalues’, ‘keys’, ‘mixed’, ‘multi’, ‘pop’, ‘popitem’, ‘setdefault’, ‘update’, ‘values’]

Patent: Portable Deaf Strobe / Vibration Personal Audio Triggered Deaf Notification Device

Sometimes I come up with an idea that I want to patent but I am so busy that I just blog it instead. Hopefully this will make it into the Internet Archive and folks can send me money when they want to license my invention.
This idea is pretty simple and my patent search (2 minutes on Google) suggests that while there are similar products, I cannot find this exact product.
The idea is to make a strobe light / vibrating system to alert deaf people when there is a particular noise from a set of noises in the room.
There are many similar products:
http://www.harriscomm.com/catalog/default.php?cPath=1215
It looks like you can nicely outfit a house for a deaf person to detect all kinds of things and signal them – even when a baby cries. But the products described above are not portable – they cannot be taken while traveling.
So my idea is a portable, self contained deaf alert system that “listens” actively for certain noises, loud speech, door knocks, crashing sounds, phone rings, etc.. It would even have a silent mode where pretty much any noise would trigger it – so while you were sleeping you could even notice a dor opening or closing.
You could even build a motion detector into the gadget to flash on motion.
The device could record audio to internal flash memory (probably only record when there is sound) – so that if something did happen – the audio could be downloaded and checked by someone who could hear. With compression and only recording when there are noises – several days would likely fit on 4GB of flash – of course there would be privacy issues with constantly recording stuff – perhaps the data could be encrypted – so special software would be needed to decrypt the audio. Maybe the audio files would need to be taken/sent/uploaded somewhere for analysis.
It might even be a good idea for non-deaf people to have such a thing – it would make them feel safer in their room while traveling.
Of course there would be some technical issues to train the gadget about which sounds to listen for – but I am betting this would yield nicely with the application of Genetic Algorithms and/or Bayesian techniques.
The gadget would need a pretty sophisticated configuration capability (although simple versions could be produced). They would likely need a USB connection to a PC for precise configuration and firmware upgrades. It would all a service technician to configure the system precisely for a less technically sophisticated device owner.
Building a prototype could be pretty easy – you just need a basic computer system with a sound digitizer and a light – frankly a One Laptop Per Child system could be used as a prototype. Perhaps some clever UM Engineering students could take this on as their project – I bet a prototype could be done in 15 weeks as long as sleep was not so important…

Saying: As a Parent, You don’t want your child’s education awarded to the low bidder

I was talking to Meghan Genovese about the new Informatics class and we were laughing about how those who predict the future of education are often off the mark. When it comes to our own education – we prefer the safe haven of bricks and mortar.
My funny retort was, “As a parent, you don’t want your child’s education awarded to the low bidder.”
Although seriously – within that notion (again as a parent) – I might award the contract for *part* of my child’s education to those who demonstrate that they teach better (regardless of cost) – and then award the rest of the contract for my child’s education to the organization which will help her be the best adult she can be and help her prepare for a diverse set of career paths and choices.
They say that one never can understand education well enough to affect the education of your own children. This is sadly true – because you only learn the right thing to do afterwards! Too bad I didn’t have like six children – I could then call the first two of them “proof of concept” and “prototype” – and after making all the mistakes on the first two – be spot on for the rest of them.

Morning Commute, Serendipity, GEOtagging Pictures, and Hot Coffee

At 7:28 AM, Lance Speelmon is hurtling through rural Indiana on Route 37 on the way to work – probably doing about 90 if I know Lance His random shuffle decides to pick a song that he and I listened to in New Orleans a long time ago – Bryan Lee and the Jump Street 5 – singing “Waiting on Ice”. Lance thinks of me and snaps this photo with his Blackberry iPhone and sends it to me:

At 7:47 AM I happen to be on line at home with a cup of coffee and see Lance’s picture/note. So I fire off a note saying hi with a smiley. Then I see the time of the note and fire off another note guessing that he is on his morning commute enroute to Indianapolis.
And then at 7:51, I idly wonder if his photo is GEOtagged. SO I pop it up in iPhoto and indeed it is and it was taken right here:

So he *was* on the way in to work – because his car was in the northbound lane.
Fun stuff.
At 8:00 AM, I went to Google Maps to use live satellite imaging to see if I could actually find where his car was on route 37. I quickly realized that we still have to wait a few years before Google puts 1000 cameras in low earth orbit to support live satellite imaging in Google Maps. So I gave up finding Lance on live satellite image (at least for now).
And then by 8:04 AM – I had written a blog post about it.
And then by 8:09 AM – I had a twitter about my blog post.
And then at 8:10 AM – I sent Lance a note with the URL of the blog post.
And then I went back to my coffee..
And it was still hot.
Update: At 8:15 Lance read my blog post and informed me that it was his iPhone 3G – so I fixed the post. But that got me thinking – does the 3G do Geotagging right when it sends pictures right from the phone… Now I finally might have a good reason to get the 3G.

Reinventing the Wheel – Stone-Simple PHP Object Relational Mapping (ORM)

Man is it fun to build stuff from scratch. About noon yesterday I was starting to write too much SQL – I have a user table that looked as shown below and I had a request object (also shown below) with lots of parameters that needed to go into the table. And I was going to have to this over and over for users, courses, tools, institutions, etc etc — Aargh. I love creative complex SQL – but Mundane SQL is not fun at all – because it is bug prone.

Friday night I asked Noah which PHP ORM he liked – we went through about 8 existing PHP ORMs (mostly over-imitating Rails) over IM and SMS. Each one he brought up – I decided it just sucked because it was too complex and required me to hack my PHP installation so bad and change my development style too much. What I wanted was effectively 80% of the features of ActiveRecord with one single PHP file that I would require_once at the beginning of my file and I could hack SQL when I wanted and use the ORM to do the nasty, dirty work.

So in the past 24 hours I developed my own PHP ORM – here are the features:

  • It does not need any hacking of PHP
  • It is unit-tested – using the unit test framework I built yesterday
  • It is one simple PHP file that you include in your source code
  • It uses ActiveRecord conventions (id, created_at, updated_at), etc.
  • It is infinitely hackable – you can step right around it if you like
  • It did foreign keys like ActiveRecord institution_id
  • It required ZERO configuration except for the table existing with some columns – no XML configuration – zip zero none – nada
  • It adapts to changes in the table trivially – if you add a column – the object knows about the column.

So here is a bit of sample code to use my ORM:

require_once("orm.php");
$usr = new ORM("user", "userid", "lti_user");
$userid = $_REQUEST[user_id];
$usr->read($userid);
$usr->setall($_REQUEST, '/^user_/');
if ( $usr->id() ) {
$usr->update();
} else {
$usr->insert($userid);
}

It is pretty Rails-like – the read() either loads the existing record or not. The setall() takes an associative array of key/value pairs (optionally limited by a regex) and updates all the fields in the object. The setall() has optional soft field matching (on by default) so the request fields don’t have to be identical to the database fields. If the object was loaded, id() exists so we update() otherwise we insert(). On the second and following ORM() call the key and table name are not needed. You can also load the ORM prior to the construction of the first model for error checking earlier.

The ORM code is 352 lines of PHP with a 152 line unit test. There is a little more work to do. But for now it works well enough for me to get back to writing my IMS LTI application without building large amounts of nasty, repetitive SQL.

The beauty of Rails and ActiveRecord is that it shows us how to build an ORM – in a sense – the ORMs of the future can assume all the thinking that went into Rails and then find new, cooler places to go – and perhaps to do so cleaner and better – and further evolve the breed.

Here is the table:

create table lti_user (
id MEDIUMINT NOT NULL AUTO_INCREMENT,
userid CHAR(255) NOT NULL,
eid CHAR(255) NULL,
displayid CHAR(255) NULL,
password CHAR(255) NULL,
firstname CHAR(255) NULL,
lastname CHAR(255) NULL,
email CHAR(255) NULL,
locale CHAR(255) NULL,
institution_id MEDIUMINT NULL,
created_at DATETIME NOT NULL,
updated_at DATETIME NOT NULL,
PRIMARY KEY (id)
);

Here is a dump of the request object:

Request Dump
Array
(
[action] => direct
[sec_digest] => 8PCloWWp4brpvuSylfyn8SaoTF4=
[sec_org_digest] => 8PCloWWp4brpvuSylfyn8SaoTF4=
[sec_nonce] => f6348bcf-87f8-11dd-b7d8-552f83eeb0b4
[sec_created] => 2008-09-21T16:18:44Z
[sec_secret] => secret
[user_id] => 0ae836b9-7fc9-4060-006f-27b2066ac545
[user_role] => Student
[user_displayid] => csev
[user_firstname] => Charles
[user_lastname] => Severance
[user_fullname] => Charles Severance
[user_email] => csev@umich.edu
[user_locale] => en_US
[user_roster] => SI300-010-F08
[course_id] => 8213060-006f-27b2066ac545
[course_code] => SI300-001-F08
[course_name] => SI300
[course_title] => Social Computing
[org_id] => umich.edu
[org_title] => University of Michigan (CTools)
[org_name] => UMich
[org_url] => https://ctools.umich.edu
[launch_resource_id] => 27b2066ac545
[launch_targets] => widget,post,iframe
[launch_resource_url] => http://www.dr-chuck.com/
[launch_tool_id] => sakai.lti.168
[launch_tool_name] => Video
[launch_tool_title] => Video Review for Midterm
[launch_width] => 320
[launch_height] => 240
[camtoolspref] =>
)

Chuck *has* learned a little bit in the past five years

I am pretty happy with myself – in the past five years – I have learned the importance of unit testing. Here I am hacking at my NG LMS based on IMS Learning Tool Interoperability – and instead of just writing a ton of code, I first spend two hours and wrote a PHP unit test framework so I could unit test all of my PHP utility code. It is pretty cool – it makes it so you don’t have to keep pressing REFRESH when you write code and also when you write debug code to exercise code – you just keep it in a unit test as well. I also wrote a PHP logging/debugging mechanism – so I don’t have to keep printing out stuff and then commenting out the prints – the logging automatically gets verbose during unit tests.

Part of this is that I miss Rails – I liked the fact that you could write less mainstream code and fall back on Rails conventions for stuff. Now you might ask – which not use a Rails clone like Ignitor or a bunch of others – the problem with those is that they are too big. I want something that gives me functionality with a single require_once() in my code – and that the distribution is a single or a few PHP scripts and not a BUNCH of hacking on my PHP installation. I want it clean and simple and a natural extension to basic PHP work patterns.

So I have progressed to understanding the importance of Unit tests but have not yet gotten to the point where I am willing to pull in a terabyte of some other framework. Ah well.

Here is sample output:

======Running Unit Tests for ALL-Tests
===Running Unit Test:ALL-Tests function:unittest_debug_01
Line 1
Line 2 <tag>
=============
Line 1
Line 2 <tag>
=============
Line 1

Line 2 <tag>
===Running Unit Test:ALL-Tests function:unittest_debug_02
Line 1
Line 2 <tag>
===Running Unit Test:ALL-Tests function:unittest_nonce_01
2008-09-20T15:22:27Z
9423067 3600
Expired
===Running Unit Test:ALL-Tests function:unittest_nonce_02
Presha1 5a10e285-6a52-4dc6-816b-d917c28302692008-06-03T13:51:20Zsecret
postsha1 zhA0wHlwZaavrEIUw0wNPlTR3PQ=
digest zhA0wHlwZaavrEIUw0wNPlTR3PQ=
Match
===Running Unit Test:ALL-Tests function:unittest_nonce_03
Presha1 5a10e285-6a52-4dc6-816b-d917c28302692008-06-03T13:51:20Zsecret
postsha1 zhA0wHlwZaavrEIUw0wNPlTR3PQ=
digest zhA0wHlwZaavrEIUw0wNPlTR3PQ=
Match
======Completed unit tests for ALL-Tests
Result: PPPPP

Saying: Rome was prototyped in a Weekend

I was talking with by buddy Rich on the phone this morning and he was teasing me about trying to write the next generation LMS in a weekend. Last weekend I wrote the client side prototype and this weekend I am writing the server side prototype.
He laughed at my silliness and said “Rome was not built in a day” – my quick retort was “Rome *was* prototyped in a weekend.” He thought it was funny – so I blogged it.