Two tools I built couldn't talk to each other

~6 min read

Stack: Python, SCORM 1.2, Quality Matters rubric, zero external dependencies for the part that matters

The setup

storycraft turns a Markdown storyboard into a SCORM package. courselint scores a SCORM package against the Quality Matters rubric - accessibility, readability, and whether the stated learning objectives are actually assessed. Two tools, same file format, both mine. I'd never run one against the other's output.

So I did. I took a real course I'd built with storycraft - a POSH (workplace harassment) compliance module, three stated objectives, three quiz questions - and pointed courselint at it.

It extracted zero objectives. Zero assessment questions. The alignment check had nothing to work with and silently skipped, and the report didn't say why - it just reported a lower coverage number and moved on.

Why

courselint's parser is built for ordinary course HTML: strip <script> tags, read visible text, regex for objective-shaped sentences and question-shaped sentences. That's a reasonable design for the courses it was built against.

storycraft's player isn't that shape. It renders everything client-side from a single embedded state object - window.__COURSE__ = {...} - with the actual slide content, objectives, and quiz answer keys sitting inside a <script> tag. Which is exactly what the parser strips first.

Neither tool is wrong on its own terms. storycraft's approach is a smaller, faster player. courselint's approach is a reasonable default for the ecosystem it was designed to audit. The gap was that I'd built both without ever checking whether the second could see the first - the kind of defect that only shows up when you actually run the pipeline end to end instead of testing each tool in isolation.

The fix

I added a second extraction path that reads the embedded state object directly when the parser's usual DOM heuristics come back empty. Instead of guessing at objectives from prose, it reads the objective bullets, quiz questions, and marked-correct answers straight out of the structured data the player already has - exact data instead of a regex hoping to spot the right sentence.

Re-ran the same course. Three objectives extracted, three assessment items, all three exactly as authored.

The part that mattered more

With real objectives and real quiz items in hand, I went looking at courselint's alignment check - the piece that's supposed to catch an objective nobody actually tests. It turned out to have no logic of its own. It was a single LLM call: send the objectives and quiz items to Claude, parse the JSON back. No API key, no check. That's a reasonable MVP, but it means the one check most directly about content quality only runs if you're willing to pay per course, and it fails closed with no fallback.

I built the fallback: a small, zero-dependency matcher that stems and compares significant words between each objective and every quiz item, and flags any objective that doesn't clear a minimum overlap. Crude compared to an LLM reading for meaning - but it costs nothing, needs no key, and runs on every course, every time, which an optional paid check structurally can't promise.

Run against the real course, it found something true: the module states three objectives - recognise harassment, distinguish complainant from respondent from the Internal Committee, and initiate a complaint through the correct channel. Two of the three have a quiz question that tests them. The third - initiate a complaint through the correct channel - is taught in the content and never assessed. Nobody checks whether a learner who finishes this course actually knows how to file a complaint.

That's a real Quality Matters 3.1 violation, in a real course I shipped, found by a hundred-odd lines of keyword matching. Coverage went from two of three rubric dimensions scored to three of three. Score dropped from a number that looked clean because a third of the rubric was silently absent, to 82 - accurate, and lower, because the tool could finally see the whole course.

Why this is in the portfolio

Most of what gets called "AI in instructional design" in 2026 is generation - Storyline exports, HeyGen avatars, ElevenLabs narration, a script from an LLM. Fewer people are building the other half: something that checks whether AI-authored or human-authored content actually teaches what it claims to, and that runs for free instead of per-call.

courselint's rule-based tier is that other half, in miniature. It's not a novel algorithm - keyword overlap with stemming is a first-year NLP exercise. What's not common is applying it specifically to the objective-versus-assessment question, wiring it in as the always-on floor under an optional LLM ceiling, and then actually running it against a real course I'd shipped instead of a synthetic fixture - which is how it found a defect a synthetic test never would have.