How to disable vmwgfx driver without compiling a new kernel on Ubuntu?

Question

I know I can disable it by compiling a new kernel without setting CONFIG_DRM_VMWGFX in config file. But I don't want build a new kernel, can I achieve it by modify some configurations?

I tried to delete vmwgfx.ko from the lib/modules/XXX/kernel/driver directory, then reboot, but the driver works well On Ubuntu 16.04. Why? Is the file elsewhere?

And I also tried to rmmod,but it's not work.

[root@ubuntu:] lsmod |grep vmwgfx
vmwgfx                235405  4 
drm_kms_helper        159169  1 vmwgfx
ttm                    99345  1 vmwgfx
drm                   370825  7 ttm,drm_kms_helper,vmwgfx
[root@ubuntu:] rmmod -f vmwgfx
rmmod: ERROR: ../libkmod/libkmod-module.c:793 kmod_module_remove_module()
could not remove 'vmwgfx': Resource temporarily unavailable
[root@ubuntu:] modprobe -r vmwgfx
modprobe: FATAL: Module vmwgfx is in use.