G’day, I’m running an Octominer with 8 x 1060s.
When you say VNC works fine but Nvidia X server does not, do you mean you’re trying to run a Nvidia specific X server rather than the default?
If the rig boots and you can PuTTY in, check the boot log to see what errors you can find with respect to the GPUs (below is my output for reference)
root@OctoMiner:~# dmesg |grep nvidia
[ 15.172632] nvidia: loading out-of-tree module taints kernel.
[ 15.172639] nvidia: module license ‘NVIDIA’ taints kernel.
[ 15.177554] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 15.184045] nvidia-nvlink: Nvlink Core is being initialized, major device number 242
[ 15.185103] nvidia 0000:01:00.0: enabling device (0000 -> 0003)
[ 15.185231] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.185327] nvidia 0000:02:00.0: enabling device (0000 -> 0003)
[ 15.185418] nvidia 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.185489] nvidia 0000:03:00.0: enabling device (0000 -> 0003)
[ 15.185568] nvidia 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.185639] nvidia 0000:05:00.0: enabling device (0000 -> 0003)
[ 15.185717] nvidia 0000:05:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.185787] nvidia 0000:06:00.0: enabling device (0000 -> 0003)
[ 15.185863] nvidia 0000:06:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.185933] nvidia 0000:07:00.0: enabling device (0000 -> 0003)
[ 15.186009] nvidia 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.186078] nvidia 0000:08:00.0: enabling device (0000 -> 0003)
[ 15.186155] nvidia 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.186223] nvidia 0000:09:00.0: enabling device (0000 -> 0003)
[ 15.186299] nvidia 0000:09:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 15.198571] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 390.25 Wed Jan 24 19:29:37 PST 2018
[ 15.200543] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[ 16.446689] nvidia-modeset: Allocated GPU:0 (GPU-ac3120e6-4a2f-ae77-71e4-f1f628784a3d) @ PCI:0000:01:00.0
[ 16.482069] [drm] [nvidia-drm] [GPU ID 0x00000200] Loading driver
[ 17.736905] nvidia-modeset: Allocated GPU:1 (GPU-9f01cddd-7f12-6203-ce39-225a83dbcb64) @ PCI:0000:02:00.0
[ 17.777651] [drm] [nvidia-drm] [GPU ID 0x00000300] Loading driver
[ 19.011558] nvidia-modeset: Allocated GPU:2 (GPU-1c1f9f39-5a8c-e66c-d505-ea6c7944f7e7) @ PCI:0000:03:00.0
[ 19.059179] [drm] [nvidia-drm] [GPU ID 0x00000500] Loading driver
[ 20.313361] nvidia-modeset: Allocated GPU:3 (GPU-e2f6ca53-d899-03dc-3827-bb6943d5480d) @ PCI:0000:05:00.0
[ 20.372474] [drm] [nvidia-drm] [GPU ID 0x00000600] Loading driver
[ 21.667746] nvidia-modeset: Allocated GPU:4 (GPU-118ba8d0-a82f-fd19-9adb-d5cb1863623c) @ PCI:0000:06:00.0
[ 21.738483] [drm] [nvidia-drm] [GPU ID 0x00000700] Loading driver
[ 22.995229] nvidia-modeset: Allocated GPU:5 (GPU-a3902ae6-1283-d86e-e033-8fabee101257) @ PCI:0000:07:00.0
[ 23.068228] [drm] [nvidia-drm] [GPU ID 0x00000800] Loading driver
[ 24.337038] nvidia-modeset: Allocated GPU:6 (GPU-afab766d-5596-3f55-133c-9beb023e906a) @ PCI:0000:08:00.0
[ 24.423594] [drm] [nvidia-drm] [GPU ID 0x00000900] Loading driver
[ 25.703751] nvidia-modeset: Allocated GPU:7 (GPU-732c51dd-2fcb-3bc3-4fe4-7a5a6edb2079) @ PCI:0000:09:00.0
[ 46.352823] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 241
Don’t worry about the “module verification failed - tainting kernel” messages, the nvidia driver isn’t open source but they’re not errors.