/usr/bin/python3.6 /home/allen/Project/deep-learning/PlanDoseQA/train.py
2022-04-22 18:19:33.442417: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Using TensorFlow backend.
2022-04-22 18:19:34.939127: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-22 18:19:34.939583: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-04-22 18:19:34.953686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:34.954103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-22 18:19:34.954117: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-22 18:19:34.955514: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-22 18:19:34.955543: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-22 18:19:34.956176: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-22 18:19:34.956318: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-22 18:19:34.957826: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-22 18:19:34.958171: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-22 18:19:34.958247: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-22 18:19:34.958304: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:34.958710: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:34.959084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-22 18:19:34.959102: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-22 18:19:35.297809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-22 18:19:35.297828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-22 18:19:35.297833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-22 18:19:35.297961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.298364: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.298735: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.299065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9883 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2022-04-22 18:19:35.299941: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-22 18:19:35.300011: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.300374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-22 18:19:35.300391: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-22 18:19:35.300410: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-22 18:19:35.300419: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-22 18:19:35.300427: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-22 18:19:35.300435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-22 18:19:35.300442: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-22 18:19:35.300450: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-22 18:19:35.300458: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-22 18:19:35.300486: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.300849: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.301196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-22 18:19:35.301242: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-22 18:19:35.301274: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/mixed_precision/loss_scale.py:56: DynamicLossScale.__init__ (from tensorflow.python.training.experimental.loss_scale) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScale
2022-04-22 18:19:35.302080: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-22 18:19:35.302124: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.302478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-22 18:19:35.302490: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-22 18:19:35.302503: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-22 18:19:35.302511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-22 18:19:35.302519: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-22 18:19:35.302527: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-22 18:19:35.302534: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-22 18:19:35.302542: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-22 18:19:35.302550: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-22 18:19:35.302578: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.302943: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.303285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-22 18:19:35.303297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-22 18:19:35.303300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-22 18:19:35.303303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-22 18:19:35.303343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.303710: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-22 18:19:35.304062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9883 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:tf.keras.mixed_precision.experimental.LossScaleOptimizer is deprecated. Please use tf.keras.mixed_precision.LossScaleOptimizer instead. Note that the non-experimental LossScaleOptimizer does not take a DynamicLossScale but instead takes the dynamic configuration directly in the constructor. For example:
  opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt)

2022-04-22 18:19:35.595350: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-04-22 18:19:35.595371: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-04-22 18:19:35.595392: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2022-04-22 18:19:35.595532: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so.11.0'; dlerror: libcupti.so.11.0: cannot open shared object file: No such file or directory
2022-04-22 18:19:35.595575: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so'; dlerror: libcupti.so: cannot open shared object file: No such file or directory
2022-04-22 18:19:35.595583: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-22 18:19:35.595593: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-04-22 18:19:35.595603: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-22 18:19:37.286469: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2022-04-22 18:19:37.304152: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3800000000 Hz
Epoch 1/1000
2022-04-22 18:19:43.768782: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-22 18:19:44.969044: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-22 18:19:45.241563: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-22 18:19:46.397956: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.65G (1774543104 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
969/969 [==============================] - 701s 714ms/step - loss: 70.9655 - ssim: 0.7729 - accuracy: 0.6464

Epoch 00001: loss improved from inf to 61.52269, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 2/1000
969/969 [==============================] - 694s 712ms/step - loss: 56.7330 - ssim: 0.8335 - accuracy: 0.6860

Epoch 00002: loss improved from 61.52269 to 55.14613, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 3/1000
969/969 [==============================] - 694s 713ms/step - loss: 51.8353 - ssim: 0.8578 - accuracy: 0.7031

Epoch 00003: loss improved from 55.14613 to 52.03955, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 4/1000
969/969 [==============================] - 693s 711ms/step - loss: 53.0182 - ssim: 0.8554 - accuracy: 0.6996

Epoch 00004: loss improved from 52.03955 to 49.48866, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 5/1000
969/969 [==============================] - 692s 711ms/step - loss: 47.3861 - ssim: 0.8707 - accuracy: 0.7017

Epoch 00005: loss improved from 49.48866 to 46.71234, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 6/1000
969/969 [==============================] - 693s 711ms/step - loss: 44.1398 - ssim: 0.8819 - accuracy: 0.7044

Epoch 00006: loss improved from 46.71234 to 43.81544, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 7/1000
969/969 [==============================] - 692s 711ms/step - loss: 40.3528 - ssim: 0.8926 - accuracy: 0.7095

Epoch 00007: loss improved from 43.81544 to 40.47824, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 8/1000
969/969 [==============================] - 692s 711ms/step - loss: 44.9492 - ssim: 0.8416 - accuracy: 0.6779

Epoch 00008: loss did not improve from 40.47824
Epoch 9/1000
969/969 [==============================] - 692s 711ms/step - loss: 35.8112 - ssim: 0.8905 - accuracy: 0.7052

Epoch 00009: loss improved from 40.47824 to 35.15625, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 10/1000
969/969 [==============================] - 692s 711ms/step - loss: 35.2400 - ssim: 0.9055 - accuracy: 0.7061

Epoch 00010: loss improved from 35.15625 to 34.81198, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 11/1000
969/969 [==============================] - 692s 711ms/step - loss: 32.7609 - ssim: 0.8953 - accuracy: 0.7031

Epoch 00011: loss improved from 34.81198 to 34.20057, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 12/1000
969/969 [==============================] - 692s 711ms/step - loss: 28.0947 - ssim: 0.9083 - accuracy: 0.7054

Epoch 00012: loss improved from 34.20057 to 30.09136, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 13/1000
969/969 [==============================] - 692s 712ms/step - loss: 28.5405 - ssim: 0.9101 - accuracy: 0.7081

Epoch 00013: loss improved from 30.09136 to 26.50772, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 14/1000
969/969 [==============================] - 692s 711ms/step - loss: 26.4600 - ssim: 0.9191 - accuracy: 0.7121

Epoch 00014: loss did not improve from 26.50772
Epoch 15/1000
969/969 [==============================] - 693s 712ms/step - loss: 25.5328 - ssim: 0.9166 - accuracy: 0.7104

Epoch 00015: loss improved from 26.50772 to 23.85921, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 16/1000
969/969 [==============================] - 694s 713ms/step - loss: 23.2277 - ssim: 0.9280 - accuracy: 0.7098

Epoch 00016: loss improved from 23.85921 to 22.80926, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 17/1000
969/969 [==============================] - 695s 714ms/step - loss: 22.9962 - ssim: 0.9266 - accuracy: 0.7099

Epoch 00017: loss improved from 22.80926 to 22.53496, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 18/1000
969/969 [==============================] - 694s 713ms/step - loss: 20.7214 - ssim: 0.9362 - accuracy: 0.7103

Epoch 00018: loss improved from 22.53496 to 20.75817, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 19/1000
969/969 [==============================] - 695s 714ms/step - loss: 18.5054 - ssim: 0.9344 - accuracy: 0.7106

Epoch 00019: loss improved from 20.75817 to 18.75440, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 20/1000
969/969 [==============================] - 694s 714ms/step - loss: 17.0368 - ssim: 0.9380 - accuracy: 0.7177

Epoch 00020: loss improved from 18.75440 to 16.69841, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 21/1000
969/969 [==============================] - 695s 714ms/step - loss: 14.6231 - ssim: 0.9414 - accuracy: 0.7210

Epoch 00021: loss improved from 16.69841 to 14.31318, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 22/1000
969/969 [==============================] - 672s 690ms/step - loss: 15.1812 - ssim: 0.9379 - accuracy: 0.7090

Epoch 00022: loss improved from 14.31318 to 13.97063, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 23/1000
969/969 [==============================] - 670s 689ms/step - loss: 13.1494 - ssim: 0.9435 - accuracy: 0.7158

Epoch 00023: loss improved from 13.97063 to 12.18239, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 24/1000
969/969 [==============================] - 671s 688ms/step - loss: 14.9581 - ssim: 0.9412 - accuracy: 0.7094

Epoch 00024: loss did not improve from 12.18239
Epoch 25/1000
969/969 [==============================] - 670s 687ms/step - loss: 11.1155 - ssim: 0.9464 - accuracy: 0.7172

Epoch 00025: loss improved from 12.18239 to 10.90919, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 26/1000
969/969 [==============================] - 669s 687ms/step - loss: 11.6154 - ssim: 0.9343 - accuracy: 0.7093

Epoch 00026: loss did not improve from 10.90919
Epoch 27/1000
969/969 [==============================] - 683s 701ms/step - loss: 8.6475 - ssim: 0.9425 - accuracy: 0.7109

Epoch 00027: loss improved from 10.90919 to 8.05492, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 28/1000
969/969 [==============================] - 695s 713ms/step - loss: 10.6213 - ssim: 0.9369 - accuracy: 0.7001

Epoch 00028: loss did not improve from 8.05492
Epoch 29/1000
969/969 [==============================] - 694s 713ms/step - loss: 7.5370 - ssim: 0.9409 - accuracy: 0.7125

Epoch 00029: loss improved from 8.05492 to 7.80014, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 30/1000
969/969 [==============================] - 694s 713ms/step - loss: 8.8589 - ssim: 0.9475 - accuracy: 0.7094

Epoch 00030: loss did not improve from 7.80014
Epoch 31/1000
969/969 [==============================] - 695s 714ms/step - loss: 6.7301 - ssim: 0.9500 - accuracy: 0.7130

Epoch 00031: loss improved from 7.80014 to 7.16307, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 32/1000
969/969 [==============================] - 694s 713ms/step - loss: 5.3361 - ssim: 0.9578 - accuracy: 0.7160

Epoch 00032: loss improved from 7.16307 to 5.23218, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 33/1000
969/969 [==============================] - 695s 714ms/step - loss: 5.9749 - ssim: 0.9494 - accuracy: 0.7137

Epoch 00033: loss did not improve from 5.23218
Epoch 34/1000
969/969 [==============================] - 695s 713ms/step - loss: 3.3936 - ssim: 0.9553 - accuracy: 0.7163

Epoch 00034: loss improved from 5.23218 to 3.86647, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 35/1000
969/969 [==============================] - 694s 713ms/step - loss: 4.5972 - ssim: 0.9445 - accuracy: 0.7083

Epoch 00035: loss did not improve from 3.86647
Epoch 36/1000
969/969 [==============================] - 695s 713ms/step - loss: 3.3686 - ssim: 0.9597 - accuracy: 0.7169

Epoch 00036: loss improved from 3.86647 to 3.18760, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 37/1000
969/969 [==============================] - 695s 714ms/step - loss: 2.5706 - ssim: 0.9587 - accuracy: 0.7157

Epoch 00037: loss did not improve from 3.18760
Epoch 38/1000
969/969 [==============================] - 696s 714ms/step - loss: 3.3538 - ssim: 0.9396 - accuracy: 0.6973

Epoch 00038: loss did not improve from 3.18760
Epoch 39/1000
969/969 [==============================] - 696s 714ms/step - loss: 2.8104 - ssim: 0.9543 - accuracy: 0.7093

Epoch 00039: loss improved from 3.18760 to 2.64508, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 40/1000
969/969 [==============================] - 695s 714ms/step - loss: 2.7283 - ssim: 0.9574 - accuracy: 0.7100

Epoch 00040: loss did not improve from 2.64508
Epoch 41/1000
969/969 [==============================] - 693s 712ms/step - loss: 2.1507 - ssim: 0.9537 - accuracy: 0.7112

Epoch 00041: loss improved from 2.64508 to 2.17889, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 42/1000
969/969 [==============================] - 694s 713ms/step - loss: 2.0846 - ssim: 0.9495 - accuracy: 0.7133

Epoch 00042: loss improved from 2.17889 to 2.00109, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 43/1000
969/969 [==============================] - 693s 712ms/step - loss: 1.8435 - ssim: 0.9548 - accuracy: 0.7177

Epoch 00043: loss improved from 2.00109 to 1.71463, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 44/1000
969/969 [==============================] - 694s 713ms/step - loss: 2.0915 - ssim: 0.9543 - accuracy: 0.7116

Epoch 00044: loss did not improve from 1.71463
Epoch 45/1000
969/969 [==============================] - 695s 713ms/step - loss: 1.7074 - ssim: 0.9580 - accuracy: 0.7205

Epoch 00045: loss did not improve from 1.71463
Epoch 46/1000
969/969 [==============================] - 694s 712ms/step - loss: 1.5083 - ssim: 0.9627 - accuracy: 0.7081

Epoch 00046: loss improved from 1.71463 to 1.39103, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 47/1000
969/969 [==============================] - 694s 713ms/step - loss: 1.0879 - ssim: 0.9652 - accuracy: 0.7170

Epoch 00047: loss improved from 1.39103 to 1.10459, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 48/1000
969/969 [==============================] - 694s 713ms/step - loss: 1.0933 - ssim: 0.9636 - accuracy: 0.7222

Epoch 00048: loss improved from 1.10459 to 1.08313, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 49/1000
969/969 [==============================] - 694s 713ms/step - loss: 1.1002 - ssim: 0.9598 - accuracy: 0.7217

Epoch 00049: loss did not improve from 1.08313
Epoch 50/1000
969/969 [==============================] - 695s 714ms/step - loss: 1.0121 - ssim: 0.9597 - accuracy: 0.7172

Epoch 00050: loss improved from 1.08313 to 0.92760, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 51/1000
969/969 [==============================] - 694s 714ms/step - loss: 0.9154 - ssim: 0.9680 - accuracy: 0.7197

Epoch 00051: loss did not improve from 0.92760
Epoch 52/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.8347 - ssim: 0.9708 - accuracy: 0.7226

Epoch 00052: loss improved from 0.92760 to 0.81608, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 53/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.9044 - ssim: 0.9596 - accuracy: 0.7213

Epoch 00053: loss improved from 0.81608 to 0.78064, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 54/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.9614 - ssim: 0.9685 - accuracy: 0.7198

Epoch 00054: loss did not improve from 0.78064
Epoch 55/1000
969/969 [==============================] - 694s 713ms/step - loss: 1.0226 - ssim: 0.9600 - accuracy: 0.7125

Epoch 00055: loss improved from 0.78064 to 0.70812, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 56/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.7548 - ssim: 0.9721 - accuracy: 0.7235

Epoch 00056: loss improved from 0.70812 to 0.64542, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 57/1000
969/969 [==============================] - 695s 714ms/step - loss: 2.2478 - ssim: 0.9646 - accuracy: 0.7208

Epoch 00057: loss did not improve from 0.64542
Epoch 58/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.9933 - ssim: 0.9621 - accuracy: 0.7119

Epoch 00058: loss did not improve from 0.64542
Epoch 59/1000
969/969 [==============================] - 695s 713ms/step - loss: 0.5067 - ssim: 0.9760 - accuracy: 0.7237

Epoch 00059: loss improved from 0.64542 to 0.55673, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 60/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.4775 - ssim: 0.9755 - accuracy: 0.7200

Epoch 00060: loss improved from 0.55673 to 0.47518, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 61/1000
969/969 [==============================] - 695s 714ms/step - loss: 0.5138 - ssim: 0.9712 - accuracy: 0.7279

Epoch 00061: loss did not improve from 0.47518
Epoch 62/1000
969/969 [==============================] - 695s 713ms/step - loss: 1.0452 - ssim: 0.9675 - accuracy: 0.7253

Epoch 00062: loss did not improve from 0.47518
Epoch 63/1000
969/969 [==============================] - 695s 714ms/step - loss: 0.6242 - ssim: 0.9695 - accuracy: 0.7249

Epoch 00063: loss did not improve from 0.47518
Epoch 64/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.4995 - ssim: 0.9758 - accuracy: 0.7324

Epoch 00064: loss improved from 0.47518 to 0.45625, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 65/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.4983 - ssim: 0.9790 - accuracy: 0.7274

Epoch 00065: loss did not improve from 0.45625
Epoch 66/1000
969/969 [==============================] - 694s 712ms/step - loss: 1.1193 - ssim: 0.9670 - accuracy: 0.7253

Epoch 00066: loss did not improve from 0.45625
Epoch 67/1000
969/969 [==============================] - 693s 712ms/step - loss: 0.5062 - ssim: 0.9736 - accuracy: 0.7286

Epoch 00067: loss did not improve from 0.45625
Epoch 68/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.4276 - ssim: 0.9770 - accuracy: 0.7253

Epoch 00068: loss did not improve from 0.45625
Epoch 69/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.3796 - ssim: 0.9709 - accuracy: 0.7228

Epoch 00069: loss improved from 0.45625 to 0.33984, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 70/1000
969/969 [==============================] - 695s 713ms/step - loss: 0.4093 - ssim: 0.9776 - accuracy: 0.7248

Epoch 00070: loss did not improve from 0.33984
Epoch 71/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.5289 - ssim: 0.9754 - accuracy: 0.7195

Epoch 00071: loss did not improve from 0.33984
Epoch 72/1000
969/969 [==============================] - 693s 711ms/step - loss: 0.4248 - ssim: 0.9746 - accuracy: 0.7173

Epoch 00072: loss did not improve from 0.33984
Epoch 73/1000
969/969 [==============================] - 694s 713ms/step - loss: 0.2854 - ssim: 0.9806 - accuracy: 0.7240

Epoch 00073: loss improved from 0.33984 to 0.29850, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 74/1000
969/969 [==============================] - 693s 712ms/step - loss: 0.6258 - ssim: 0.9795 - accuracy: 0.7257

Epoch 00074: loss did not improve from 0.29850
Epoch 75/1000
969/969 [==============================] - 695s 713ms/step - loss: 0.4669 - ssim: 0.9784 - accuracy: 0.7302

Epoch 00075: loss did not improve from 0.29850
Epoch 76/1000
969/969 [==============================] - 693s 711ms/step - loss: 0.2953 - ssim: 0.9811 - accuracy: 0.7340

Epoch 00076: loss improved from 0.29850 to 0.29456, saving model to ./model/model_8_mse_relu_v1.hdf5
Epoch 77/1000
969/969 [==============================] - 693s 712ms/step - loss: 0.4823 - ssim: 0.9812 - accuracy: 0.7291

Epoch 00077: loss did not improve from 0.29456
Epoch 78/1000
969/969 [==============================] - 694s 712ms/step - loss: 0.5369 - ssim: 0.9816 - accuracy: 0.7242

Epoch 00078: loss did not improve from 0.29456
Epoch 79/1000
969/969 [==============================] - 693s 711ms/step - loss: 0.5703 - ssim: 0.9771 - accuracy: 0.7262

Epoch 00079: loss did not improve from 0.29456
Epoch 80/1000
969/969 [==============================] - 693s 712ms/step - loss: 0.4551 - ssim: 0.9811 - accuracy: 0.7215

Epoch 00080: loss did not improve from 0.29456
Epoch 81/1000
969/969 [==============================] - 693s 712ms/step - loss: 0.3637 - ssim: 0.9816 - accuracy: 0.7240

Epoch 00081: loss did not improve from 0.29456
Epoch 82/1000
969/969 [==============================] - 692s 711ms/step - loss: 0.3844 - ssim: 0.9800 - accuracy: 0.7256

Epoch 00082: loss did not improve from 0.29456
Epoch 83/1000
228/969 [======>.......................] - ETA: 8:47 - loss: 0.3385 - ssim: 0.9835 - accuracy: 0.7299Traceback (most recent call last):
  File "/home/allen/Project/deep-learning/PlanDoseQA/train.py", line 46, in <module>
    use_multiprocessing=False, workers=10, max_queue_size=10, callbacks=[checkpoint])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 855, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2943, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
KeyboardInterrupt
2022-04-23 10:09:12.934173: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]
terminate called without an active exception

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
s