Detected 9 NVIDIA cards
GPU BUS ID : 05 06 09 0a 0b 0c 0d 0e 0f
CLOCK :
MEM :
PLIMIT :
FAN :
NVS query error:
=== GPU 0, 05:00.0 Tesla K10.G1.8GB 3527 MB, PL: 85 W, 117.5 W, 125 W === 00:58:56
SET POWER LIMIT: 117.5 W
Max Perf mode: 3
(exicode=1)
=== GPU 1, 06:00.0 Tesla K10.G1.8GB 3527 MB, PL: 85 W, 117.5 W, 125 W === 00:59:14
SET POWER LIMIT: 117.5 W
Max Perf mode: 3
(exicode=1)
=== GPU 2, 09:00.0 Tesla K80 11441 MB, PL: 100 W, 149 W, 175 W === 00:59:26
SET POWER LIMIT: 149.0 W
Max Perf mode: 3
(exicode=1)
=== GPU 3, 0a:00.0 Tesla K80 11441 MB, PL: 100 W, 149 W, 175 W === 00:59:38
SET POWER LIMIT: 149.0 W
Max Perf mode: 3
(exicode=1)
=== GPU 4, 0b:00.0 GeForce GTX 1070 Ti 8119 MB, PL: 90 W, 180 W, 217 W === 00:59:51
SET POWER LIMIT: 180.0 W
Max Perf mode: 3
(exicode=1)
=== GPU 5, 0c:00.0 GeForce GTX 1070 Ti 8119 MB, PL: 90 W, 180 W, 217 W === 01:00:03
SET POWER LIMIT: 180.0 W
Max Perf mode: 3
(exicode=1)
=== GPU 6, 0d:00.0 GeForce GTX TITAN 6083 MB, PL: 150 W, 250 W, 265 W === 01:00:15
SET POWER LIMIT: 250.0 W
Max Perf mode: 3
(exicode=1)
=== GPU 7, 0e:00.0 GeForce GTX TITAN 6083 MB, PL: 150 W, 250 W, 265 W === 01:00:27
SET POWER LIMIT: 250.0 W
Max Perf mode: 3
(exicode=1)
=== GPU 8, 0f:00.0 GeForce GTX TITAN Black 6083 MB, PL: 150 W, 250 W, 265 W === 01:00:40
SET POWER LIMIT: 250.0 W [Insufficient Resources]
(exicode=23)
Max Perf mode: 3