Short answer: no. Slowness is a result of how a site is built, not WordPress itself.
You often read that a WordPress site takes three to six seconds to load, even after optimization. That is true for a site with a heavy theme, twenty plugins, and images that no one has compressed. That describes most WordPress sites, so it is a fair observation. However, it is not a characteristic of the platform.
In a recent client project, we went from 4.8 to 1.2 seconds, with all three Core Web Vitals in the green. The same CMS, but with a short plugin list, optimized images, and correct caching.
A site without a CMS remains lighter, that is true. The question is whether that last half-second is worth more to you than being able to access your own content.