I've figured it out!
Turns out that when recording the action, the JPG quality during Save is not, well.... saved. The quality it uses after recording is whatever you chose last time you saw that dialog box. So when manually saving and choosing, e.g., 6, the action would use 6. Then, when I wanted a higher quality pic for some other reason and chose 12, the action would use 12 (and give me much larger file sizes).
I definitely think having Photoshop remember the quality during Save should be part of action recording, and I hope Adobe changes this in the future.