To uninstall xvfb just follow these instructions.
sudo apt-get remove xvfb
This will remove just the xvfb package itself.
sudo apt-get remove --auto-remove xvfb
This will remove the xvfb package and any other dependant packages which are no longer needed.
If you also want to delete your local/config files for xvfb then this will work.
sudo apt-get purge xvfb
Or similarly, like this xvfb
sudo apt-get purge --auto-remove xvfb
|Maintainer||Debian X Strike Force <email@example.com>|
|Description||Virtual Framebuffer 'fake' X server and no physical input devices. It emulates a dumb framebuffer using virtual memory. The primary use of this server was intended to be server testing, but other novel uses for it have been found, including testing clients against unusual depths and screen configurations, doing batch processing with Xvfb as a background rendering engine, load testing, as an aid to porting the X server to a new platform, and providing an unobtrusive way to run applications that don't really need an X server but insist on having one anyway.|