Adobe has researched these slowdown issues, and fixed every one we can reproduce. But many of the issues only happen on specific systems, and frequently get tracked down to something specific about that system outside of Photoshop itself (antivirus, firewall, bad drives, bad drivers, other applications messing with the OS, etc.).
If Photoshop is grabbing RAM, it is because it does need it for something that you opened. That could be a document, or it could be presets that you have loaded. By default, Photoshop does not use that much RAM. And Photoshop won't use that much RAM until you load or do something that needs that much RAM.
Yes, even if no files are open, Photoshop will hold onto the RAM it has allocated - because it is MUCH, MUCH faster for Photoshop to manage it's RAM than to deallocate it and wait for the OS to reallocate it. (this is a pattern you will find in most applications that use a lot of RAM, including video and imaging applications and databases) There is no need to shutdown Photoshop to free memory - the memory it has will be reused for whatever you load or do next. Photoshop will also free memory if the OS needs additional memory (due to paging or hints). Photoshop has used the same memory management since version 2.0.
If Photoshop's memory usage is slowing down your system somehow, then you may need more RAM for the work that you are doing. Or you may have some other applications that are not managing their own RAM usage very well.
Now, we have seen some third party plugins that have memory leaks - and those are beyond our control.
But we test Photoshop rather heavily for memory leaks, and fix them rapidly.