/usr/bin/python3.6 /home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py
2022-04-26 18:07:07.629247: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Using TensorFlow backend.
2022-04-26 18:07:09.283231: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 18:07:09.283701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-04-26 18:07:09.296955: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.297351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-26 18:07:09.297369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 18:07:09.298806: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 18:07:09.298835: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 18:07:09.299478: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-26 18:07:09.299615: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-26 18:07:09.301224: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-26 18:07:09.301581: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-26 18:07:09.301665: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 18:07:09.301716: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.302111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.302466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-26 18:07:09.302486: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 18:07:09.632645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-26 18:07:09.632669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-26 18:07:09.632674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-26 18:07:09.632808: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.633209: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.633578: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.633937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9755 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2022-04-26 18:07:09.635160: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 18:07:09.635221: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.635580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-26 18:07:09.635598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 18:07:09.635621: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 18:07:09.635632: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 18:07:09.635642: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-26 18:07:09.635652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-26 18:07:09.635662: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-26 18:07:09.635672: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-26 18:07:09.635682: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 18:07:09.635711: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.636083: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.636420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-26 18:07:09.636469: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 18:07:09.636501: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/mixed_precision/loss_scale.py:56: DynamicLossScale.__init__ (from tensorflow.python.training.experimental.loss_scale) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScale
2022-04-26 18:07:09.637297: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-26 18:07:09.637344: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.637697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-26 18:07:09.637708: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-26 18:07:09.637720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 18:07:09.637729: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 18:07:09.637737: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-26 18:07:09.637745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-26 18:07:09.637753: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-26 18:07:09.637761: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-26 18:07:09.637769: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 18:07:09.637797: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.638160: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.638499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-26 18:07:09.638512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-26 18:07:09.638516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-26 18:07:09.638519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-26 18:07:09.638560: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.638924: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-26 18:07:09.639269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9755 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:tf.keras.mixed_precision.experimental.LossScaleOptimizer is deprecated. Please use tf.keras.mixed_precision.LossScaleOptimizer instead. Note that the non-experimental LossScaleOptimizer does not take a DynamicLossScale but instead takes the dynamic configuration directly in the constructor. For example:
  opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt)

2022-04-26 18:07:10.015441: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-04-26 18:07:10.015465: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-04-26 18:07:10.015485: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2022-04-26 18:07:10.015617: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so.11.0'; dlerror: libcupti.so.11.0: cannot open shared object file: No such file or directory
2022-04-26 18:07:10.015658: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so'; dlerror: libcupti.so: cannot open shared object file: No such file or directory
2022-04-26 18:07:10.015669: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-26 18:07:10.015680: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-04-26 18:07:10.015690: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-26 18:07:12.040999: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2022-04-26 18:07:12.060165: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3800000000 Hz
Epoch 1/100
2022-04-26 18:07:18.648790: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-26 18:07:19.942276: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-26 18:07:20.219400: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-26 18:07:21.393149: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.53G (1640493056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  1/969 [..............................] - ETA: 2:54:00 - loss: 2.0623e-04 - ssim: 0.9997 - accuracy: 0.71312022-04-26 18:07:23.097615: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-04-26 18:07:23.097632: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-04-26 18:07:23.097657: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
  2/969 [..............................] - ETA: 15:45 - loss: 1.8086e-04 - ssim: 0.9997 - accuracy: 0.7187  2022-04-26 18:07:23.834541: I tensorflow/core/profiler/lib/profiler_session.cc:71] Profiler session collecting data.
2022-04-26 18:07:23.834638: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-26 18:07:23.856032: I tensorflow/core/profiler/internal/gpu/cupti_collector.cc:228]  GpuTracer has collected 0 callback api events and 0 activity events.
2022-04-26 18:07:23.864053: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-04-26 18:07:23.869443: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: ../log/train/plugins/profile/2022_04_26_18_07_23
2022-04-26 18:07:23.870494: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for trace.json.gz to ../log/train/plugins/profile/2022_04_26_18_07_23/Server2080ti.trace.json.gz
2022-04-26 18:07:23.898531: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: ../log/train/plugins/profile/2022_04_26_18_07_23
2022-04-26 18:07:23.903061: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for memory_profile.json.gz to ../log/train/plugins/profile/2022_04_26_18_07_23/Server2080ti.memory_profile.json.gz
2022-04-26 18:07:23.903577: I tensorflow/core/profiler/rpc/client/capture_profile.cc:251] Creating directory: ../log/train/plugins/profile/2022_04_26_18_07_23Dumped tool data for xplane.pb to ../log/train/plugins/profile/2022_04_26_18_07_23/Server2080ti.xplane.pb
Dumped tool data for overview_page.pb to ../log/train/plugins/profile/2022_04_26_18_07_23/Server2080ti.overview_page.pb
Dumped tool data for input_pipeline.pb to ../log/train/plugins/profile/2022_04_26_18_07_23/Server2080ti.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to ../log/train/plugins/profile/2022_04_26_18_07_23/Server2080ti.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to ../log/train/plugins/profile/2022_04_26_18_07_23/Server2080ti.kernel_stats.pb

969/969 [==============================] - 684s 696ms/step - loss: 0.0012 - ssim: 0.9981 - accuracy: 0.7340

Epoch 00001: loss improved from inf to 0.00103, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 2/100
969/969 [==============================] - 690s 709ms/step - loss: 5.7750e-04 - ssim: 0.9991 - accuracy: 0.7253

Epoch 00002: loss improved from 0.00103 to 0.00070, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 3/100
969/969 [==============================] - 693s 712ms/step - loss: 6.2029e-04 - ssim: 0.9990 - accuracy: 0.7298

Epoch 00003: loss did not improve from 0.00070
Epoch 4/100
969/969 [==============================] - 693s 712ms/step - loss: 6.9213e-04 - ssim: 0.9989 - accuracy: 0.7371

Epoch 00004: loss did not improve from 0.00070
Epoch 5/100
969/969 [==============================] - 691s 711ms/step - loss: 8.4749e-04 - ssim: 0.9986 - accuracy: 0.7379

Epoch 00005: loss improved from 0.00070 to 0.00060, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 6/100
969/969 [==============================] - 691s 710ms/step - loss: 3.3475e-04 - ssim: 0.9994 - accuracy: 0.7364

Epoch 00006: loss improved from 0.00060 to 0.00042, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 7/100
969/969 [==============================] - 691s 710ms/step - loss: 2.1337e-04 - ssim: 0.9997 - accuracy: 0.7377

Epoch 00007: loss improved from 0.00042 to 0.00032, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 8/100
969/969 [==============================] - 691s 711ms/step - loss: 1.6826e-04 - ssim: 0.9997 - accuracy: 0.7393

Epoch 00008: loss did not improve from 0.00032
Epoch 9/100
969/969 [==============================] - 692s 711ms/step - loss: 6.4617e-04 - ssim: 0.9990 - accuracy: 0.7330

Epoch 00009: loss did not improve from 0.00032
Epoch 10/100
969/969 [==============================] - 691s 711ms/step - loss: 3.7251e-04 - ssim: 0.9995 - accuracy: 0.7354

Epoch 00010: loss did not improve from 0.00032
Epoch 11/100
969/969 [==============================] - 692s 711ms/step - loss: 5.8225e-04 - ssim: 0.9991 - accuracy: 0.7311

Epoch 00011: loss did not improve from 0.00032
Epoch 12/100
969/969 [==============================] - 692s 711ms/step - loss: 5.0695e-04 - ssim: 0.9992 - accuracy: 0.7341

Epoch 00012: loss did not improve from 0.00032
Epoch 13/100
969/969 [==============================] - 691s 711ms/step - loss: 2.8111e-04 - ssim: 0.9996 - accuracy: 0.7311

Epoch 00013: loss did not improve from 0.00032
Epoch 14/100
969/969 [==============================] - 692s 711ms/step - loss: 3.3508e-04 - ssim: 0.9995 - accuracy: 0.7289

Epoch 00014: loss did not improve from 0.00032
Epoch 15/100
969/969 [==============================] - 692s 711ms/step - loss: 6.8516e-04 - ssim: 0.9990 - accuracy: 0.7383

Epoch 00015: loss did not improve from 0.00032
Epoch 16/100
969/969 [==============================] - 692s 711ms/step - loss: 4.6525e-04 - ssim: 0.9993 - accuracy: 0.7337

Epoch 00016: loss did not improve from 0.00032
Epoch 17/100
969/969 [==============================] - 694s 713ms/step - loss: 5.8587e-04 - ssim: 0.9991 - accuracy: 0.7386

Epoch 00017: loss did not improve from 0.00032
Epoch 18/100
969/969 [==============================] - 693s 712ms/step - loss: 1.2609e-04 - ssim: 0.9998 - accuracy: 0.7355

Epoch 00018: loss improved from 0.00032 to 0.00022, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 19/100
969/969 [==============================] - 693s 713ms/step - loss: 4.2999e-04 - ssim: 0.9993 - accuracy: 0.7338

Epoch 00019: loss did not improve from 0.00022
Epoch 20/100
969/969 [==============================] - 694s 714ms/step - loss: 3.3983e-04 - ssim: 0.9995 - accuracy: 0.7375

Epoch 00020: loss did not improve from 0.00022
Epoch 21/100
969/969 [==============================] - 692s 712ms/step - loss: 5.7099e-04 - ssim: 0.9992 - accuracy: 0.7298

Epoch 00021: loss did not improve from 0.00022
Epoch 22/100
969/969 [==============================] - 694s 713ms/step - loss: 9.7674e-04 - ssim: 0.9985 - accuracy: 0.7322

Epoch 00022: loss did not improve from 0.00022
Epoch 23/100
969/969 [==============================] - 694s 713ms/step - loss: 5.5932e-04 - ssim: 0.9992 - accuracy: 0.7340

Epoch 00023: loss did not improve from 0.00022
Epoch 24/100
969/969 [==============================] - 693s 713ms/step - loss: 5.6804e-04 - ssim: 0.9991 - accuracy: 0.7319

Epoch 00024: loss did not improve from 0.00022
Epoch 25/100
969/969 [==============================] - 694s 713ms/step - loss: 1.5641e-04 - ssim: 0.9998 - accuracy: 0.7448

Epoch 00025: loss did not improve from 0.00022
Epoch 26/100
969/969 [==============================] - 693s 713ms/step - loss: 2.0153e-04 - ssim: 0.9997 - accuracy: 0.7380

Epoch 00026: loss did not improve from 0.00022
Epoch 27/100
969/969 [==============================] - 693s 713ms/step - loss: 3.2446e-04 - ssim: 0.9995 - accuracy: 0.7398

Epoch 00027: loss did not improve from 0.00022
Epoch 28/100
969/969 [==============================] - 693s 712ms/step - loss: 6.4997e-04 - ssim: 0.9990 - accuracy: 0.7327

Epoch 00028: loss did not improve from 0.00022
Epoch 29/100
969/969 [==============================] - 694s 713ms/step - loss: 2.8344e-04 - ssim: 0.9996 - accuracy: 0.7290

Epoch 00029: loss did not improve from 0.00022
Epoch 30/100
969/969 [==============================] - 694s 713ms/step - loss: 5.3734e-04 - ssim: 0.9991 - accuracy: 0.7348

Epoch 00030: loss did not improve from 0.00022
Epoch 31/100
969/969 [==============================] - 693s 712ms/step - loss: 5.2161e-04 - ssim: 0.9992 - accuracy: 0.7335

Epoch 00031: loss did not improve from 0.00022
Epoch 32/100
969/969 [==============================] - 694s 713ms/step - loss: 3.1433e-04 - ssim: 0.9995 - accuracy: 0.7413

Epoch 00032: loss did not improve from 0.00022
Epoch 33/100
969/969 [==============================] - 693s 713ms/step - loss: 7.0748e-04 - ssim: 0.9989 - accuracy: 0.7351

Epoch 00033: loss did not improve from 0.00022
Epoch 34/100
969/969 [==============================] - 693s 712ms/step - loss: 3.6202e-04 - ssim: 0.9994 - accuracy: 0.7401

Epoch 00034: loss did not improve from 0.00022
Epoch 35/100
969/969 [==============================] - 692s 712ms/step - loss: 3.5225e-04 - ssim: 0.9995 - accuracy: 0.7336

Epoch 00035: loss did not improve from 0.00022
Epoch 36/100
969/969 [==============================] - 694s 713ms/step - loss: 2.7441e-04 - ssim: 0.9995 - accuracy: 0.7389

Epoch 00036: loss did not improve from 0.00022
Epoch 37/100
969/969 [==============================] - 693s 713ms/step - loss: 1.1188e-04 - ssim: 0.9998 - accuracy: 0.7392

Epoch 00037: loss improved from 0.00022 to 0.00019, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 38/100
969/969 [==============================] - 693s 713ms/step - loss: 5.4489e-04 - ssim: 0.9992 - accuracy: 0.7304

Epoch 00038: loss did not improve from 0.00019
Epoch 39/100
969/969 [==============================] - 693s 713ms/step - loss: 5.0226e-04 - ssim: 0.9992 - accuracy: 0.7308

Epoch 00039: loss did not improve from 0.00019
Epoch 40/100
969/969 [==============================] - 694s 713ms/step - loss: 5.1992e-04 - ssim: 0.9992 - accuracy: 0.7258

Epoch 00040: loss did not improve from 0.00019
Epoch 41/100
969/969 [==============================] - 694s 714ms/step - loss: 7.9393e-04 - ssim: 0.9987 - accuracy: 0.7434

Epoch 00041: loss did not improve from 0.00019
Epoch 42/100
969/969 [==============================] - 695s 714ms/step - loss: 6.4558e-04 - ssim: 0.9990 - accuracy: 0.7372

Epoch 00042: loss did not improve from 0.00019
Epoch 43/100
969/969 [==============================] - 694s 713ms/step - loss: 8.9758e-04 - ssim: 0.9987 - accuracy: 0.7332

Epoch 00043: loss did not improve from 0.00019
Epoch 44/100
969/969 [==============================] - 694s 714ms/step - loss: 3.9704e-04 - ssim: 0.9994 - accuracy: 0.7353

Epoch 00044: loss did not improve from 0.00019
Epoch 45/100
969/969 [==============================] - 694s 713ms/step - loss: 1.1481e-04 - ssim: 0.9998 - accuracy: 0.7341

Epoch 00045: loss improved from 0.00019 to 0.00017, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 46/100
969/969 [==============================] - 693s 713ms/step - loss: 6.4016e-04 - ssim: 0.9991 - accuracy: 0.7328

Epoch 00046: loss did not improve from 0.00017
Epoch 47/100
969/969 [==============================] - 694s 714ms/step - loss: 7.9355e-04 - ssim: 0.9988 - accuracy: 0.7338

Epoch 00047: loss did not improve from 0.00017
Epoch 48/100
969/969 [==============================] - 694s 713ms/step - loss: 2.0853e-04 - ssim: 0.9997 - accuracy: 0.7276

Epoch 00048: loss did not improve from 0.00017
Epoch 49/100
969/969 [==============================] - 693s 712ms/step - loss: 4.1749e-04 - ssim: 0.9994 - accuracy: 0.7317

Epoch 00049: loss did not improve from 0.00017
Epoch 50/100
969/969 [==============================] - 693s 713ms/step - loss: 3.3277e-04 - ssim: 0.9995 - accuracy: 0.7342

Epoch 00050: loss did not improve from 0.00017
Epoch 51/100
969/969 [==============================] - 693s 713ms/step - loss: 1.4088e-04 - ssim: 0.9998 - accuracy: 0.7421

Epoch 00051: loss did not improve from 0.00017
Epoch 52/100
969/969 [==============================] - 693s 713ms/step - loss: 1.9327e-04 - ssim: 0.9997 - accuracy: 0.7341

Epoch 00052: loss did not improve from 0.00017
Epoch 53/100
969/969 [==============================] - 694s 714ms/step - loss: 7.7931e-04 - ssim: 0.9988 - accuracy: 0.7274

Epoch 00053: loss did not improve from 0.00017
Epoch 54/100
969/969 [==============================] - 694s 713ms/step - loss: 6.2170e-04 - ssim: 0.9990 - accuracy: 0.7335

Epoch 00054: loss did not improve from 0.00017
Epoch 55/100
969/969 [==============================] - 693s 713ms/step - loss: 4.2895e-04 - ssim: 0.9994 - accuracy: 0.7372

Epoch 00055: loss did not improve from 0.00017
Epoch 56/100
969/969 [==============================] - 694s 713ms/step - loss: 4.4574e-04 - ssim: 0.9993 - accuracy: 0.7339

Epoch 00056: loss did not improve from 0.00017
Epoch 57/100
969/969 [==============================] - 693s 712ms/step - loss: 1.6372e-04 - ssim: 0.9997 - accuracy: 0.7354

Epoch 00057: loss did not improve from 0.00017
Epoch 58/100
969/969 [==============================] - 692s 712ms/step - loss: 4.8463e-04 - ssim: 0.9993 - accuracy: 0.7409

Epoch 00058: loss did not improve from 0.00017
Epoch 59/100
969/969 [==============================] - 693s 712ms/step - loss: 5.7576e-05 - ssim: 0.9999 - accuracy: 0.7341

Epoch 00059: loss improved from 0.00017 to 0.00012, saving model to ../model/model_8_mse_sigmoid_v2.hdf5
Epoch 60/100
969/969 [==============================] - 692s 712ms/step - loss: 2.3881e-04 - ssim: 0.9996 - accuracy: 0.7356

Epoch 00060: loss did not improve from 0.00012
Epoch 61/100
969/969 [==============================] - 693s 712ms/step - loss: 5.7853e-04 - ssim: 0.9991 - accuracy: 0.7349

Epoch 00061: loss did not improve from 0.00012
Epoch 62/100
969/969 [==============================] - 692s 712ms/step - loss: 4.0141e-04 - ssim: 0.9994 - accuracy: 0.7335

Epoch 00062: loss did not improve from 0.00012
Epoch 63/100
969/969 [==============================] - 692s 711ms/step - loss: 2.0486e-04 - ssim: 0.9997 - accuracy: 0.7345

Epoch 00063: loss did not improve from 0.00012
Epoch 64/100
969/969 [==============================] - 693s 712ms/step - loss: 3.7822e-04 - ssim: 0.9994 - accuracy: 0.7248

Epoch 00064: loss did not improve from 0.00012
Epoch 65/100
969/969 [==============================] - 692s 712ms/step - loss: 2.5976e-04 - ssim: 0.9997 - accuracy: 0.7308

Epoch 00065: loss did not improve from 0.00012
Epoch 66/100
969/969 [==============================] - 693s 712ms/step - loss: 4.5061e-04 - ssim: 0.9993 - accuracy: 0.7296

Epoch 00066: loss did not improve from 0.00012
Epoch 67/100
969/969 [==============================] - 693s 712ms/step - loss: 2.2152e-04 - ssim: 0.9997 - accuracy: 0.7397

Epoch 00067: loss did not improve from 0.00012
Epoch 68/100
969/969 [==============================] - 692s 712ms/step - loss: 3.0416e-04 - ssim: 0.9995 - accuracy: 0.7346

Epoch 00068: loss did not improve from 0.00012
Epoch 69/100
969/969 [==============================] - 692s 712ms/step - loss: 2.2279e-04 - ssim: 0.9997 - accuracy: 0.7360

Epoch 00069: loss did not improve from 0.00012
Epoch 70/100
969/969 [==============================] - 692s 712ms/step - loss: 2.7806e-04 - ssim: 0.9996 - accuracy: 0.7323

Epoch 00070: loss did not improve from 0.00012
Epoch 71/100
969/969 [==============================] - 692s 711ms/step - loss: 1.6226e-04 - ssim: 0.9998 - accuracy: 0.7355

Epoch 00071: loss did not improve from 0.00012
Epoch 72/100
969/969 [==============================] - 693s 712ms/step - loss: 2.2820e-04 - ssim: 0.9997 - accuracy: 0.7336

Epoch 00072: loss did not improve from 0.00012
Epoch 73/100
969/969 [==============================] - 693s 712ms/step - loss: 3.0774e-04 - ssim: 0.9996 - accuracy: 0.7320

Epoch 00073: loss did not improve from 0.00012
Epoch 74/100
969/969 [==============================] - 693s 712ms/step - loss: 1.6853e-04 - ssim: 0.9998 - accuracy: 0.7349

Epoch 00074: loss did not improve from 0.00012
Epoch 75/100
969/969 [==============================] - 693s 712ms/step - loss: 4.4689e-04 - ssim: 0.9994 - accuracy: 0.7292

Epoch 00075: loss did not improve from 0.00012
Epoch 76/100
969/969 [==============================] - 692s 712ms/step - loss: 1.6332e-04 - ssim: 0.9998 - accuracy: 0.7401

Epoch 00076: loss did not improve from 0.00012
Epoch 77/100
969/969 [==============================] - 692s 712ms/step - loss: 6.3284e-04 - ssim: 0.9990 - accuracy: 0.7390

Epoch 00077: loss did not improve from 0.00012
Epoch 78/100
969/969 [==============================] - 687s 706ms/step - loss: 2.1990e-04 - ssim: 0.9997 - accuracy: 0.7315

Epoch 00078: loss did not improve from 0.00012
Epoch 79/100
969/969 [==============================] - 668s 686ms/step - loss: 3.8360e-04 - ssim: 0.9994 - accuracy: 0.7331

Epoch 00079: loss did not improve from 0.00012
Epoch 80/100
969/969 [==============================] - 668s 686ms/step - loss: 1.5908e-04 - ssim: 0.9998 - accuracy: 0.7427

Epoch 00080: loss did not improve from 0.00012
Epoch 81/100
969/969 [==============================] - 668s 687ms/step - loss: 2.4949e-04 - ssim: 0.9996 - accuracy: 0.7340

Epoch 00081: loss did not improve from 0.00012
Epoch 82/100
969/969 [==============================] - 668s 687ms/step - loss: 1.3693e-04 - ssim: 0.9998 - accuracy: 0.7353

Epoch 00082: loss did not improve from 0.00012
Epoch 83/100
969/969 [==============================] - 668s 686ms/step - loss: 2.1324e-04 - ssim: 0.9997 - accuracy: 0.7308

Epoch 00083: loss did not improve from 0.00012
Epoch 84/100
969/969 [==============================] - 668s 687ms/step - loss: 1.6769e-04 - ssim: 0.9997 - accuracy: 0.7303

Epoch 00084: loss did not improve from 0.00012
Epoch 85/100
969/969 [==============================] - 668s 686ms/step - loss: 2.1734e-04 - ssim: 0.9997 - accuracy: 0.7405

Epoch 00085: loss did not improve from 0.00012
Epoch 86/100
969/969 [==============================] - 668s 687ms/step - loss: 1.9619e-04 - ssim: 0.9997 - accuracy: 0.7349

Epoch 00086: loss did not improve from 0.00012
Epoch 87/100
969/969 [==============================] - 672s 691ms/step - loss: 2.7037e-04 - ssim: 0.9996 - accuracy: 0.7337

Epoch 00087: loss did not improve from 0.00012
Epoch 88/100
969/969 [==============================] - 671s 689ms/step - loss: 2.3737e-04 - ssim: 0.9996 - accuracy: 0.7365

Epoch 00088: loss did not improve from 0.00012
Epoch 89/100
 12/969 [..............................] - ETA: 11:21 - loss: 6.3452e-05 - ssim: 0.9999 - accuracy: 0.7095Traceback (most recent call last):
  File "/home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py", line 47, in <module>
    use_multiprocessing=False, workers=10, max_queue_size=10, callbacks=[checkpoint, board])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 855, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2943, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
KeyboardInterrupt
terminate called without an active exception

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
