koaning.io

I am copying a pattern from Simon Willison by also having Claude Code write me some notebooks on occasion about algorithm-peformance things that I'm curious about. A few years ago, it probably would not have been worth it for me to make this investment, but these days ...

The topic of todays exercise is GridSearchCV from scikit-learn and ways it could be so much faster if you designed it to work for specific use-cases. Under the hood it uses joblib/pickle to serialise scikit-learn pipelines and this comes at a cost.

I had Claude write me a notebook to compare a few different approaches for linear models, specifically.

Ridge Regression

Ridge is a linear model that comes with a regularisation parameter that you might want to loop over. Here's the thing though; you can pick an optimiser that leverages that Ridge has a closed form solution.

$$ \mathbf{w} = (\mathbf{X}^T \mathbf{X} + \alpha \mathbf{I})^{-1} \mathbf{X}^T \mathbf{y} $$

You can even write your NumPy code in a clever way so that you don't have to loop over all the $\alpha$-values. You can just figure out all the optimal weights on one swoop. If you compare it to looping over lots of calls to Ridge().fit or, even worse, doing that inside of a GridSearchCV you're gonna get some overhead.

Logistic Regression

Logistic regression does not have a closed form solution, but you can apply another trick. After you've trained your first model, you can consider using those weights as a starting point to train the next model that has a slightly different regularisation parameter. This is also a massive speedup! One that's easy to perform when you write the code "by hand" but not something that grid-search can do for you because it assumes that all

Pramatic?

Should you rewrite all your code now? No! Scikit-learn tries to solve an utmost general problem, there's bound to be many more of these instances where you can get better specific performance.

Is it an interesting lesson, though? One that's easier to observe thanks to new tools? Sure!

Is this a free lunch? No, not at all. Claude did a bunch of boilerplate right but it did get a lot of important nuance wrong. It originally made comparisons where it didn't just measure the joblib overhead but also performed cross-validations which isn't a fair comparison. You still need to check the work of the LLM, even if it speeds up the boilerplate and lowers the barrier of entry for these sorts of things.

Notebook for this work can be found here.

Math can be so interesting sometimes.

Let's say you want to simplify this:

$$ \frac{\sin^2 x}{1 - \cos x} - 1 $$

One path

I thought, aha!, there's a 1 there! And I know that $\sin^2x + \cos^2 = 1 $. So I could just do this right?

$$ \frac{\sin^2 x}{1 - \cos x} - \sin^2x + \cos^2 $$

And from here you could expand and try something like:

$$ \frac{\sin^2 x}{1 - \cos x} - \frac{(1 - \cos x)(\sin^2x + \cos^2)}{1 - \cos x} $$

You now have something with the same denominator and this would get you there eventually ... but only if you're completely sure that you don't make any manual mistakes.

Another path

Instead you could also rewrite $\sin^2x + \cos^2 = 1 $ into $\cos^2 x = 1 - \sin^2x $. Why would this help? Well let's go back to the start.

$$ \frac{\sin^2 x}{1 - \cos x} - 1 $$

This becomes.

$$ = \frac{1 - \cos^2 x}{1 - \cos x} - 1 $$

You can factor this.

$$ = \frac{(1 - \cos x)(1 + \cos x)}{1 - \cos x} - 1 $$

And hey! Notice how things cancel out pretty early here!

$$ = \frac{(\cancel{1 - \cos x})(1 + \cos x)}{\cancel{1 - \cos x}} - 1 $$

That makes things a lot simpler.

$$ = 1 + \cos x - 1 $$

$$ = \cos x $$

The tricky part with treating math as an occasional hobby, instead of a daily activity, is that you get very rusty at detecting the right path early.

Most of the time when I need autocompletion on the terminal I really just need it for files and folders. With zsh this works out of the box when you use most known linux commands, but chains of Python commands can totally flunk here.

Enter compdef, so for marimo I added the following two lines to my ~/.zshrc:

# Enable file completion for uvx commands
compdef _default uvx

# Add uvx marimo alias
alias mam="uvx marimo -y edit --sandbox --watch"

This adds a nice marimo alias but it also makes sure that the alias gets the file autocompletion.