NVidia display drivers are drivers for 2D/3D graphics. No Display Drivers – no 3D games.
NVidia PhysX are drivers for game physics – all GeForce cards above GeForce 8400 support hardware acceleration for physX physics simulations (along with support for CUDA), but physX system software can provide support for physics even if your GPU doesn’t support hardware acceleration for physics. PhysX is getting more popular lately and seem to replace Havoc in many games – because it is free to use. No PhysX drivers – no games that use PhysX for physics (and that’s quite a lot). As far as I know physx driver interacts with display driver, but they are still separate components – because GPU (i.e. videocard) acceleration for physics can be enabled/disabled and isn’t present in all cards.
> Or should I keep both?
You should keep both.

keep both physx is used in games and your driver is used to display what your viewing, if you dont play any games AT ALL then its probably safe to delete the physx

Nvidia purchased Ageia’s PhysX engine and they are now implementing physics hardware acceleration/processing into their GeForce 8 series and above video cards. I would leave it, it may increase performance in certain games that support it.
Originally the Ageia’s PhysX was a physcial add-in card, but since Nvidia’s acquired it, they have implemented it into their newer device drivers. They are off loading the physics calculations to the GPU instead of having the CPU handle them.