/usr/bin/python3.6 /home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py
2022-04-25 19:12:36.256048: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Using TensorFlow backend.
2022-04-25 19:12:37.754939: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 19:12:37.755397: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-04-25 19:12:37.769218: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:37.769613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-25 19:12:37.769627: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 19:12:37.771015: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 19:12:37.771042: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 19:12:37.771657: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-25 19:12:37.771792: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-25 19:12:37.773396: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-25 19:12:37.773747: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-25 19:12:37.773823: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 19:12:37.773873: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:37.774272: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:37.774636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-25 19:12:37.774656: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 19:12:38.099690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-25 19:12:38.099714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-25 19:12:38.099718: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-25 19:12:38.099846: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.100263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.100637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.100997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9845 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2022-04-25 19:12:38.102219: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 19:12:38.102275: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.102637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-25 19:12:38.102652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 19:12:38.102671: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 19:12:38.102680: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 19:12:38.102687: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-25 19:12:38.102695: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-25 19:12:38.102702: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-25 19:12:38.102710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-25 19:12:38.102718: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 19:12:38.102745: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.103110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.103454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-25 19:12:38.103502: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 19:12:38.103533: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/mixed_precision/loss_scale.py:56: DynamicLossScale.__init__ (from tensorflow.python.training.experimental.loss_scale) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScale
2022-04-25 19:12:38.104360: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 19:12:38.104403: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.104756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-25 19:12:38.104767: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 19:12:38.104779: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 19:12:38.104789: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 19:12:38.104798: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-25 19:12:38.104807: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-25 19:12:38.104816: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-25 19:12:38.104825: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-25 19:12:38.104834: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 19:12:38.104863: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.105229: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.105572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-25 19:12:38.105584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-25 19:12:38.105587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-25 19:12:38.105590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-25 19:12:38.105629: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.106055: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 19:12:38.106433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9845 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:tf.keras.mixed_precision.experimental.LossScaleOptimizer is deprecated. Please use tf.keras.mixed_precision.LossScaleOptimizer instead. Note that the non-experimental LossScaleOptimizer does not take a DynamicLossScale but instead takes the dynamic configuration directly in the constructor. For example:
  opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt)

2022-04-25 19:12:38.490192: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-04-25 19:12:38.490214: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-04-25 19:12:38.490233: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2022-04-25 19:12:38.490381: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so.11.0'; dlerror: libcupti.so.11.0: cannot open shared object file: No such file or directory
2022-04-25 19:12:38.490427: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so'; dlerror: libcupti.so: cannot open shared object file: No such file or directory
2022-04-25 19:12:38.490434: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-25 19:12:38.490446: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-04-25 19:12:38.490456: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-25 19:12:40.289004: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2022-04-25 19:12:40.308159: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3800000000 Hz
Epoch 1/100
2022-04-25 19:12:47.409184: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 19:12:48.625859: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 19:12:48.903169: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 19:12:50.090917: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.62G (1735116544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
969/969 [==============================] - 706s 718ms/step - loss: 0.6393 - ssim: 0.9784 - accuracy: 0.7307

Epoch 00001: loss improved from inf to 0.98319, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 2/100
969/969 [==============================] - 700s 718ms/step - loss: 1.0090 - ssim: 0.9754 - accuracy: 0.7260

Epoch 00002: loss improved from 0.98319 to 0.94193, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 3/100
969/969 [==============================] - 700s 719ms/step - loss: 1.0196 - ssim: 0.9746 - accuracy: 0.7258

Epoch 00003: loss improved from 0.94193 to 0.65982, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 4/100
969/969 [==============================] - 700s 719ms/step - loss: 0.3637 - ssim: 0.9820 - accuracy: 0.7323

Epoch 00004: loss improved from 0.65982 to 0.40466, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 5/100
969/969 [==============================] - 700s 718ms/step - loss: 0.8129 - ssim: 0.9777 - accuracy: 0.7336

Epoch 00005: loss did not improve from 0.40466
Epoch 6/100
969/969 [==============================] - 701s 719ms/step - loss: 1.0111 - ssim: 0.9774 - accuracy: 0.7269

Epoch 00006: loss did not improve from 0.40466
Epoch 7/100
969/969 [==============================] - 700s 718ms/step - loss: 0.9084 - ssim: 0.9801 - accuracy: 0.7281

Epoch 00007: loss did not improve from 0.40466
Epoch 8/100
969/969 [==============================] - 700s 718ms/step - loss: 0.3863 - ssim: 0.9839 - accuracy: 0.7324

Epoch 00008: loss improved from 0.40466 to 0.36071, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 9/100
969/969 [==============================] - 700s 719ms/step - loss: 0.4573 - ssim: 0.9856 - accuracy: 0.7336

Epoch 00009: loss did not improve from 0.36071
Epoch 10/100
969/969 [==============================] - 701s 719ms/step - loss: 1.0243 - ssim: 0.9724 - accuracy: 0.7219

Epoch 00010: loss did not improve from 0.36071
Epoch 11/100
969/969 [==============================] - 703s 722ms/step - loss: 0.3393 - ssim: 0.9844 - accuracy: 0.7342

Epoch 00011: loss did not improve from 0.36071
Epoch 12/100
969/969 [==============================] - 701s 720ms/step - loss: 0.4969 - ssim: 0.9852 - accuracy: 0.7350

Epoch 00012: loss did not improve from 0.36071
Epoch 13/100
969/969 [==============================] - 703s 721ms/step - loss: 0.7561 - ssim: 0.9680 - accuracy: 0.7285

Epoch 00013: loss did not improve from 0.36071
Epoch 14/100
969/969 [==============================] - 703s 721ms/step - loss: 0.3588 - ssim: 0.9868 - accuracy: 0.7360

Epoch 00014: loss improved from 0.36071 to 0.34981, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 15/100
969/969 [==============================] - 703s 722ms/step - loss: 0.4861 - ssim: 0.9842 - accuracy: 0.7333

Epoch 00015: loss did not improve from 0.34981
Epoch 16/100
969/969 [==============================] - 703s 721ms/step - loss: 0.3441 - ssim: 0.9809 - accuracy: 0.7422

Epoch 00016: loss did not improve from 0.34981
Epoch 17/100
969/969 [==============================] - 703s 721ms/step - loss: 0.5412 - ssim: 0.9861 - accuracy: 0.7343

Epoch 00017: loss did not improve from 0.34981
Epoch 18/100
969/969 [==============================] - 702s 720ms/step - loss: 1.2778 - ssim: 0.9786 - accuracy: 0.7383

Epoch 00018: loss did not improve from 0.34981
Epoch 19/100
969/969 [==============================] - 702s 720ms/step - loss: 1.4071 - ssim: 0.9721 - accuracy: 0.7353

Epoch 00019: loss did not improve from 0.34981
Epoch 20/100
969/969 [==============================] - 702s 720ms/step - loss: 0.6151 - ssim: 0.9814 - accuracy: 0.7337

Epoch 00020: loss did not improve from 0.34981
Epoch 21/100
969/969 [==============================] - 702s 721ms/step - loss: 0.2954 - ssim: 0.9852 - accuracy: 0.7337

Epoch 00021: loss did not improve from 0.34981
Epoch 22/100
969/969 [==============================] - 702s 721ms/step - loss: 0.3471 - ssim: 0.9879 - accuracy: 0.7389

Epoch 00022: loss improved from 0.34981 to 0.29838, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 23/100
969/969 [==============================] - 702s 721ms/step - loss: 0.8017 - ssim: 0.9801 - accuracy: 0.7296

Epoch 00023: loss did not improve from 0.29838
Epoch 24/100
969/969 [==============================] - 702s 720ms/step - loss: 0.7478 - ssim: 0.9791 - accuracy: 0.7313

Epoch 00024: loss did not improve from 0.29838
Epoch 25/100
969/969 [==============================] - 702s 720ms/step - loss: 0.5715 - ssim: 0.9817 - accuracy: 0.7241

Epoch 00025: loss did not improve from 0.29838
Epoch 26/100
969/969 [==============================] - 702s 720ms/step - loss: 0.7934 - ssim: 0.9780 - accuracy: 0.7374

Epoch 00026: loss did not improve from 0.29838
Epoch 27/100
969/969 [==============================] - 702s 720ms/step - loss: 0.4708 - ssim: 0.9813 - accuracy: 0.7252

Epoch 00027: loss did not improve from 0.29838
Epoch 28/100
969/969 [==============================] - 702s 721ms/step - loss: 0.2755 - ssim: 0.9893 - accuracy: 0.7385

Epoch 00028: loss improved from 0.29838 to 0.26953, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 29/100
969/969 [==============================] - 702s 720ms/step - loss: 0.2899 - ssim: 0.9871 - accuracy: 0.7331

Epoch 00029: loss did not improve from 0.26953
Epoch 30/100
969/969 [==============================] - 702s 720ms/step - loss: 0.4455 - ssim: 0.9868 - accuracy: 0.7306

Epoch 00030: loss did not improve from 0.26953
Epoch 31/100
969/969 [==============================] - 703s 721ms/step - loss: 1.0575 - ssim: 0.9806 - accuracy: 0.7263

Epoch 00031: loss did not improve from 0.26953
Epoch 32/100
969/969 [==============================] - 701s 719ms/step - loss: 0.3318 - ssim: 0.9794 - accuracy: 0.7362

Epoch 00032: loss did not improve from 0.26953
Epoch 33/100
969/969 [==============================] - 702s 720ms/step - loss: 0.2745 - ssim: 0.9854 - accuracy: 0.7301

Epoch 00033: loss did not improve from 0.26953
Epoch 34/100
969/969 [==============================] - 702s 720ms/step - loss: 0.3828 - ssim: 0.9855 - accuracy: 0.7283

Epoch 00034: loss did not improve from 0.26953
Epoch 35/100
969/969 [==============================] - 702s 720ms/step - loss: 0.3306 - ssim: 0.9865 - accuracy: 0.7338

Epoch 00035: loss did not improve from 0.26953
Epoch 36/100
969/969 [==============================] - 702s 720ms/step - loss: 0.5338 - ssim: 0.9806 - accuracy: 0.7302

Epoch 00036: loss did not improve from 0.26953
Epoch 37/100
969/969 [==============================] - 702s 720ms/step - loss: 0.8935 - ssim: 0.9786 - accuracy: 0.7283

Epoch 00037: loss did not improve from 0.26953
Epoch 38/100
969/969 [==============================] - 702s 720ms/step - loss: 0.4298 - ssim: 0.9862 - accuracy: 0.7300

Epoch 00038: loss did not improve from 0.26953
Epoch 39/100
969/969 [==============================] - 703s 721ms/step - loss: 0.2343 - ssim: 0.9871 - accuracy: 0.7292

Epoch 00039: loss did not improve from 0.26953
Epoch 40/100
969/969 [==============================] - 703s 722ms/step - loss: 0.4166 - ssim: 0.9845 - accuracy: 0.7265

Epoch 00040: loss did not improve from 0.26953
Epoch 41/100
969/969 [==============================] - 704s 722ms/step - loss: 0.2323 - ssim: 0.9886 - accuracy: 0.7273

Epoch 00041: loss improved from 0.26953 to 0.22532, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 42/100
969/969 [==============================] - 703s 721ms/step - loss: 0.3292 - ssim: 0.9866 - accuracy: 0.7287

Epoch 00042: loss did not improve from 0.22532
Epoch 43/100
969/969 [==============================] - 702s 720ms/step - loss: 0.9652 - ssim: 0.9716 - accuracy: 0.7265

Epoch 00043: loss did not improve from 0.22532
Epoch 44/100
969/969 [==============================] - 702s 720ms/step - loss: 0.4318 - ssim: 0.9864 - accuracy: 0.7270

Epoch 00044: loss did not improve from 0.22532
Epoch 45/100
969/969 [==============================] - 701s 720ms/step - loss: 0.8989 - ssim: 0.9755 - accuracy: 0.7305

Epoch 00045: loss did not improve from 0.22532
Epoch 46/100
969/969 [==============================] - 703s 721ms/step - loss: 0.3765 - ssim: 0.9835 - accuracy: 0.7295

Epoch 00046: loss did not improve from 0.22532
Epoch 47/100
969/969 [==============================] - 702s 720ms/step - loss: 0.2061 - ssim: 0.9890 - accuracy: 0.7311

Epoch 00047: loss did not improve from 0.22532
Epoch 48/100
969/969 [==============================] - 702s 720ms/step - loss: 0.6406 - ssim: 0.9842 - accuracy: 0.7255

Epoch 00048: loss did not improve from 0.22532
Epoch 49/100
969/969 [==============================] - 701s 719ms/step - loss: 0.3265 - ssim: 0.9899 - accuracy: 0.7321

Epoch 00049: loss did not improve from 0.22532
Epoch 50/100
969/969 [==============================] - 703s 720ms/step - loss: 0.3418 - ssim: 0.9891 - accuracy: 0.7348

Epoch 00050: loss did not improve from 0.22532
Epoch 51/100
969/969 [==============================] - 704s 722ms/step - loss: 0.2922 - ssim: 0.9898 - accuracy: 0.7350

Epoch 00051: loss did not improve from 0.22532
Epoch 52/100
969/969 [==============================] - 702s 720ms/step - loss: 0.1952 - ssim: 0.9902 - accuracy: 0.7335

Epoch 00052: loss improved from 0.22532 to 0.21828, saving model to ../model/model_8_3_mse_relu_v2.hdf5
Epoch 53/100
969/969 [==============================] - 702s 721ms/step - loss: 0.3991 - ssim: 0.9885 - accuracy: 0.7295

Epoch 00053: loss did not improve from 0.21828
Epoch 54/100
969/969 [==============================] - 702s 721ms/step - loss: 0.2499 - ssim: 0.9900 - accuracy: 0.7362

Epoch 00054: loss did not improve from 0.21828
Epoch 55/100
969/969 [==============================] - 703s 721ms/step - loss: 0.4041 - ssim: 0.9864 - accuracy: 0.7382

Epoch 00055: loss did not improve from 0.21828
Epoch 56/100
969/969 [==============================] - 702s 720ms/step - loss: 0.2627 - ssim: 0.9902 - accuracy: 0.7348

Epoch 00056: loss did not improve from 0.21828
Epoch 57/100
969/969 [==============================] - 701s 719ms/step - loss: 0.3055 - ssim: 0.9889 - accuracy: 0.7280

Epoch 00057: loss did not improve from 0.21828
Epoch 58/100
969/969 [==============================] - 700s 719ms/step - loss: 0.3957 - ssim: 0.9779 - accuracy: 0.7239

Epoch 00058: loss did not improve from 0.21828
Epoch 59/100
969/969 [==============================] - 700s 718ms/step - loss: 1.0622 - ssim: 0.9612 - accuracy: 0.7281

Epoch 00059: loss did not improve from 0.21828
Epoch 60/100
969/969 [==============================] - 701s 719ms/step - loss: 0.7771 - ssim: 0.9638 - accuracy: 0.7166

Epoch 00060: loss did not improve from 0.21828
Epoch 61/100
969/969 [==============================] - 701s 719ms/step - loss: 0.5031 - ssim: 0.9663 - accuracy: 0.7190

Epoch 00061: loss did not improve from 0.21828
Epoch 62/100
969/969 [==============================] - 700s 718ms/step - loss: 0.4770 - ssim: 0.9699 - accuracy: 0.7317

Epoch 00062: loss did not improve from 0.21828
Epoch 63/100
969/969 [==============================] - 701s 719ms/step - loss: 0.6922 - ssim: 0.9669 - accuracy: 0.7336

Epoch 00063: loss did not improve from 0.21828
Epoch 64/100
969/969 [==============================] - 701s 719ms/step - loss: 0.6846 - ssim: 0.9680 - accuracy: 0.7288

Epoch 00064: loss did not improve from 0.21828
Epoch 65/100
969/969 [==============================] - 701s 719ms/step - loss: 0.8840 - ssim: 0.9627 - accuracy: 0.7367

Epoch 00065: loss did not improve from 0.21828
Epoch 66/100
969/969 [==============================] - 701s 719ms/step - loss: 0.5162 - ssim: 0.9631 - accuracy: 0.7378

Epoch 00066: loss did not improve from 0.21828
Epoch 67/100
969/969 [==============================] - 700s 718ms/step - loss: 0.5444 - ssim: 0.9595 - accuracy: 0.7241

Epoch 00067: loss did not improve from 0.21828
Epoch 68/100
969/969 [==============================] - 701s 719ms/step - loss: 1.1732 - ssim: 0.9553 - accuracy: 0.7352

Epoch 00068: loss did not improve from 0.21828
Epoch 69/100
969/969 [==============================] - 701s 719ms/step - loss: 1.2121 - ssim: 0.9494 - accuracy: 0.7241

Epoch 00069: loss did not improve from 0.21828
Epoch 70/100
969/969 [==============================] - 701s 719ms/step - loss: 0.8421 - ssim: 0.9596 - accuracy: 0.7320

Epoch 00070: loss did not improve from 0.21828
Epoch 71/100
969/969 [==============================] - 701s 719ms/step - loss: 0.7777 - ssim: 0.9627 - accuracy: 0.7294

Epoch 00071: loss did not improve from 0.21828
Epoch 72/100
969/969 [==============================] - 684s 702ms/step - loss: 0.7007 - ssim: 0.9593 - accuracy: 0.7285

Epoch 00072: loss did not improve from 0.21828
Epoch 73/100
 71/969 [=>............................] - ETA: 10:47 - loss: 1.7977 - ssim: 0.9330 - accuracy: 0.7339Traceback (most recent call last):
  File "/home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py", line 47, in <module>
    use_multiprocessing=False, workers=10, max_queue_size=10, callbacks=[checkpoint])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 855, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2943, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
KeyboardInterrupt
2022-04-26 09:15:29.795150: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]

Process finished with exit code 1
