So I haven't been playing Skyrim for a while. I've continued to look over mods and such and add some in. I think I inadvertently removed a mod or two as well. It was just something to do while I was working from home.
In any case, there's some danger in removing mods. Apparently, even though you remove the mod, there are scripts left orphaned in your save that can still impact the game. In my case, this was causing the game to stop for a second or two every 5 minutes until an eventual crash. It also was steadily increasing the file size of my save by a megabyte every 5 minutes or so.
I managed to find a save a bit back where this phenomenon didn't occur. Oddly enough, it seems to stop right at my save before starting out the mod Moonpath to Elswyr. I had gotten frustrated with the mod and removed it. I had managed to play several hours without it. Apparently some other mod later on triggered toxic behavior from its orphaned scripts. I did a lot of reading and there's just no way to clean this kind of thing up.
So some good rules:
1) Always keep "landmark" saves, in case you need to restore.
2) Beware of mods that involve quests, behaviors, and events. If you add them in, you should commit to them for the life of that character.
Now I gotta go do the stupid Dark Brotherhood quest line again.