Quick update on the rasterizer thing. I got the a rotating Stanford bunny (~70k tris) rendering at 10-15 FPS (~30-35 FPS if static).
I think this is as far as I'll go for now.
By the way, for those that want to try to implement something like this, the implementation was heavily based on https://jtsorlinis.github.io/rendering-tutorial/
A very short and sweet tutorial on rasterization, with some nice interactive demos.