does it make sense to server-push woff2?

Well that's an interesting question alright. The answer is: No you should not do this. But the reason is a little different than you might think...

For reasons that are a bit cryptic, fonts are always requested without credentials (basically cookies). For most browsers (Edge being the exception) this means the browser opens another connection for that request and this is important because HTTP/2 Pushes are linked to the connection. So if you push a resource on one connection, and the browser goes to get a resource from another connection it will not use that pushed resource (you do not push directly into the HTTP Cache as you might think).

This, and lots of other HTTP/2 Push trickiness and edge cases were discussed by Jake Archibald in his excellent HTTP/2 push is tougher than I thought article.

But it does beg the question of how you can decide what format to push even if this wasn't an issue, or if you wanted to send different image formats for example (that would be on the same connection). Other than looking at the User-Agent and guessing based off of that, there is now way for you to know what the browser supports.

There is a new HTTP Client Hints header currently being proposed which aims to allow the browser to indicate the device specifics. This currently is more concerned with image size and density, but could in theory also include the file formats that are supported.