Yeah, such pc would mostly work. Assuming we talk about amd64 platform.
- cpu is backward compatible, and could run kernel from the 90s, be that linux or windows.
- gpu can also run in VGA compatibility mode
- usb can run in compatibility mode - but usb appeared in 1996.
- output from gpu would be difficult, it would be best to bring gpu with VGA output. There was no hdmi until like 2003
- ethernet phys can usually work in 10mbit compatibility mode as well
You would not be able to achieve full potential of the hardware due to missing drivers, GPU would still probably be faster, but not nearly as fast as it could be with drivers. Same goes to CPU but to lesser effect. But no software could use new sse instructions for example. And I think multi core would not work, meaning you would have single core cpu.
And if you can run kernel, then you can run almost every software for that time, since software doesn't care about hardware. It cares about kernel and its ABI.
[edit]
Oh yeah, one thing that would probably blow up. SATA/nvme. It does not exist. There are no drivers for that, meaning you can't use drives. So you better bring some ide to pci to pcie controllers xD
Also 4gb of ram is max what you could use. PAE in linux was implemented near year 2000. But that's not an issues since in those time you could literally fit everything inside ryzen L3 cache xD