Believe it or not, there are still a lot of places you can get MP3s if you're willing to look around. Amazon still sells them, although sometimes you have to do some extra clicking in order to get past the "listen via streaming" crap they put up on every page.
I also like to go to thrift stores and pick up cheap CDs to rip onto my PC. Nobody really wants them much, and you have to dig through all the crap, but you can still get some good finds.
Discogs.com also has a second hand marketplace that usually has really good prices on most albums.
If you're into JPop, here's a blog with a ton of downloads:
https://jpop80ss2.blogspot.com/Archive.org has tons of weird rips from cassettes and old vinyls:
https://archive.org/details/cassettetapes?tab=collectionThere are also game soundtracks:
https://archive.org/details/folksoundomy_gamesoundtracks?&sort=-week&page=2Free Music Archive is slowly becoming more corporate, but they still have free music downloads:
https://freemusicarchive.org/homeBandcamp still offers downloads for (almost) all of their albums. A good number of artists have free albums as well. You can sort by free stuff here:
https://bandcamp.com/tag/free?tab=all_releasesI'm a bit of an mp3 hoarder myself. I keep most of my music on an SD card in my phone and listen locally rather than streaming. I'm picky about what I listen to and I like the freedom of being able to listen to things I love regardless of whether or not I'm online.