/usr/bin/python3.6 /home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py
2022-04-26 16:36:13.801701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Using TensorFlow backend.
2022-04-26 16:36:15.461914: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 16:36:15.462374: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-04-26 16:36:15.472282: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.472633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-26 16:36:15.472647: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 16:36:15.474048: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 16:36:15.474076: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 16:36:15.474693: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-26 16:36:15.474833: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-26 16:36:15.476409: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-26 16:36:15.476752: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-26 16:36:15.476829: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 16:36:15.476877: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.477255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.477602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-26 16:36:15.477619: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 16:36:15.800228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-26 16:36:15.800252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-26 16:36:15.800257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-26 16:36:15.800385: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.800757: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.801125: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.801454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9755 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2022-04-26 16:36:15.802348: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 16:36:15.802406: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.802767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-26 16:36:15.802785: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 16:36:15.802803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 16:36:15.802813: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 16:36:15.802822: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-26 16:36:15.802830: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-26 16:36:15.802839: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-26 16:36:15.802847: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-26 16:36:15.802856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 16:36:15.802889: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.803257: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.803600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-26 16:36:15.803649: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 16:36:15.803682: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/mixed_precision/loss_scale.py:56: DynamicLossScale.__init__ (from tensorflow.python.training.experimental.loss_scale) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScale
2022-04-26 16:36:15.804513: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 16:36:15.804559: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.804910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-26 16:36:15.804921: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 16:36:15.804933: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 16:36:15.804942: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 16:36:15.804950: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-26 16:36:15.804958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-26 16:36:15.804966: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-26 16:36:15.804974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-26 16:36:15.804983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 16:36:15.805013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.805365: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.805708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-26 16:36:15.805720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-26 16:36:15.805724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-26 16:36:15.805727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-26 16:36:15.805768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.806129: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 16:36:15.806470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9755 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:tf.keras.mixed_precision.experimental.LossScaleOptimizer is deprecated. Please use tf.keras.mixed_precision.LossScaleOptimizer instead. Note that the non-experimental LossScaleOptimizer does not take a DynamicLossScale but instead takes the dynamic configuration directly in the constructor. For example:
  opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt)

2022-04-26 16:36:16.109715: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-04-26 16:36:16.109736: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-04-26 16:36:16.109755: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2022-04-26 16:36:16.109893: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so.11.0'; dlerror: libcupti.so.11.0: cannot open shared object file: No such file or directory
2022-04-26 16:36:16.109934: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so'; dlerror: libcupti.so: cannot open shared object file: No such file or directory
2022-04-26 16:36:16.109945: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-26 16:36:16.109955: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-04-26 16:36:16.109965: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-26 16:36:17.762855: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2022-04-26 16:36:17.780134: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3800000000 Hz
Epoch 1/100
2022-04-26 16:36:24.454425: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 16:36:25.595367: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 16:36:25.870547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 16:36:27.096391: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.53G (1640493056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  1/969 [..............................] - ETA: 2:53:46 - loss: 0.3477 - ssim: 0.5456 - accuracy: 0.14002022-04-26 16:36:28.798021: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-04-26 16:36:28.798045: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-04-26 16:36:28.798072: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
  2/969 [..............................] - ETA: 15:42 - loss: 0.3030 - ssim: 0.5995 - accuracy: 0.2620  2022-04-26 16:36:29.537894: I tensorflow/core/profiler/lib/profiler_session.cc:71] Profiler session collecting data.
2022-04-26 16:36:29.537990: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-26 16:36:29.561153: I tensorflow/core/profiler/internal/gpu/cupti_collector.cc:228]  GpuTracer has collected 0 callback api events and 0 activity events.
2022-04-26 16:36:29.569163: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-04-26 16:36:29.575796: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: ../log/train/plugins/profile/2022_04_26_16_36_29
2022-04-26 16:36:29.576820: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for trace.json.gz to ../log/train/plugins/profile/2022_04_26_16_36_29/Server2080ti.trace.json.gz
2022-04-26 16:36:29.605275: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: ../log/train/plugins/profile/2022_04_26_16_36_29
2022-04-26 16:36:29.609529: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for memory_profile.json.gz to ../log/train/plugins/profile/2022_04_26_16_36_29/Server2080ti.memory_profile.json.gz
2022-04-26 16:36:29.610036: I tensorflow/core/profiler/rpc/client/capture_profile.cc:251] Creating directory: ../log/train/plugins/profile/2022_04_26_16_36_29Dumped tool data for xplane.pb to ../log/train/plugins/profile/2022_04_26_16_36_29/Server2080ti.xplane.pb
Dumped tool data for overview_page.pb to ../log/train/plugins/profile/2022_04_26_16_36_29/Server2080ti.overview_page.pb
Dumped tool data for input_pipeline.pb to ../log/train/plugins/profile/2022_04_26_16_36_29/Server2080ti.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to ../log/train/plugins/profile/2022_04_26_16_36_29/Server2080ti.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to ../log/train/plugins/profile/2022_04_26_16_36_29/Server2080ti.kernel_stats.pb

969/969 [==============================] - 690s 702ms/step - loss: 0.0159 - ssim: 0.9767 - accuracy: 0.7329

Epoch 00001: loss improved from inf to 0.00485, saving model to ../model/model_8_mse_sigmoid_v1.hdf5
Epoch 2/100
969/969 [==============================] - 683s 703ms/step - loss: 0.0020 - ssim: 0.9969 - accuracy: 0.7323

Epoch 00002: loss improved from 0.00485 to 0.00122, saving model to ../model/model_8_mse_sigmoid_v1.hdf5
Epoch 3/100
969/969 [==============================] - 671s 690ms/step - loss: 0.0013 - ssim: 0.9980 - accuracy: 0.7398

Epoch 00003: loss did not improve from 0.00122
Epoch 4/100
969/969 [==============================] - 671s 690ms/step - loss: 9.1120e-04 - ssim: 0.9986 - accuracy: 0.7379

Epoch 00004: loss improved from 0.00122 to 0.00094, saving model to ../model/model_8_mse_sigmoid_v1.hdf5
Epoch 5/100
969/969 [==============================] - 679s 697ms/step - loss: 6.5071e-04 - ssim: 0.9990 - accuracy: 0.7382

Epoch 00005: loss improved from 0.00094 to 0.00072, saving model to ../model/model_8_mse_sigmoid_v1.hdf5
Epoch 6/100
969/969 [==============================] - 668s 687ms/step - loss: 7.6606e-04 - ssim: 0.9988 - accuracy: 0.7324

Epoch 00006: loss did not improve from 0.00072
Epoch 7/100
117/969 [==>...........................] - ETA: 10:03 - loss: 3.2144e-04 - ssim: 0.9994 - accuracy: 0.7232Traceback (most recent call last):
  File "/home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py", line 47, in <module>
    use_multiprocessing=False, workers=10, max_queue_size=10, callbacks=[checkpoint, board])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 855, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2943, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
KeyboardInterrupt
2022-04-26 17:45:36.840910: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]

Process finished with exit code 1
