It sounds like it’s the kernel but whether it has anything to do with ssh, I really don’t know. Sometimes parts work together in surprising ways, as I learned with the recent sshd/systemd/xz exploit.

You might be fine and this was the most alarming exploit since it’s very inconvenient, but personally I’d restart just to be sure.

No - it’s the kernel image - the actual operating system, rather than a service that runs on top of it.

If you just want to restart your ssh service after updating the packages, then “systemctl restart sshd” is all that’s needed, although you should probably reboot whenever the package manager suggests as a general good habit.