In that case only reboot will fix it
If your kde4 session last very long, than strange memory leaks may happen.
Also, if some application is accessing lots of data on hard drive it may force kernel to move inactive memory to swap.
This also mean that you need to provide more useful information.