/usr/bin/python3.6 /home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py
2022-04-25 09:30:12.558070: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Using TensorFlow backend.
2022-04-25 09:30:14.193656: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 09:30:14.194240: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-04-25 09:30:14.205636: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.206034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-25 09:30:14.206051: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 09:30:14.207551: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 09:30:14.207595: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 09:30:14.208491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-25 09:30:14.208752: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-25 09:30:14.211112: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-25 09:30:14.211672: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-25 09:30:14.211804: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 09:30:14.211901: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.212408: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.212812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-25 09:30:14.212835: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 09:30:14.660552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-25 09:30:14.660584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-25 09:30:14.660590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-25 09:30:14.660782: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.661308: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.661738: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.662080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9836 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2022-04-25 09:30:14.662926: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 09:30:14.663000: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.663404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-25 09:30:14.663429: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 09:30:14.663461: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 09:30:14.663470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 09:30:14.663479: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-25 09:30:14.663487: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-25 09:30:14.663495: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-25 09:30:14.663503: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-25 09:30:14.663511: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 09:30:14.663558: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.663936: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.664358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-25 09:30:14.664451: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 09:30:14.664488: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/mixed_precision/loss_scale.py:56: DynamicLossScale.__init__ (from tensorflow.python.training.experimental.loss_scale) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScale
2022-04-25 09:30:14.665400: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-04-25 09:30:14.665459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.665866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-04-25 09:30:14.665882: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-25 09:30:14.665903: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 09:30:14.665914: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 09:30:14.665925: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-25 09:30:14.665936: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-25 09:30:14.665947: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-04-25 09:30:14.665958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-25 09:30:14.665969: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 09:30:14.666008: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.666426: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.666815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-04-25 09:30:14.666839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-25 09:30:14.666845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2022-04-25 09:30:14.666850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2022-04-25 09:30:14.666907: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.667333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-25 09:30:14.667803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9836 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:tf.keras.mixed_precision.experimental.LossScaleOptimizer is deprecated. Please use tf.keras.mixed_precision.LossScaleOptimizer instead. Note that the non-experimental LossScaleOptimizer does not take a DynamicLossScale but instead takes the dynamic configuration directly in the constructor. For example:
  opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt)

2022-04-25 09:30:15.012283: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-04-25 09:30:15.012303: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-04-25 09:30:15.012324: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2022-04-25 09:30:15.012470: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so.11.0'; dlerror: libcupti.so.11.0: cannot open shared object file: No such file or directory
2022-04-25 09:30:15.012515: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcupti.so'; dlerror: libcupti.so: cannot open shared object file: No such file or directory
2022-04-25 09:30:15.012524: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-25 09:30:15.012534: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-04-25 09:30:15.012544: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-04-25 09:30:16.932926: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2022-04-25 09:30:16.952086: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3800000000 Hz
Epoch 1/100
2022-04-25 09:30:24.513420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-25 09:30:25.719606: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-25 09:30:25.996533: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-04-25 09:30:27.180863: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.61G (1725260032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
969/969 [==============================] - 692s 703ms/step - loss: 43.2283 - ssim: 0.7682 - accuracy: 0.5961

Epoch 00001: loss improved from inf to 23.24214, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 2/100
969/969 [==============================] - 678s 696ms/step - loss: 8.7346 - ssim: 0.9083 - accuracy: 0.6881

Epoch 00002: loss improved from 23.24214 to 7.50084, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 3/100
969/969 [==============================] - 678s 696ms/step - loss: 5.6083 - ssim: 0.9346 - accuracy: 0.7032

Epoch 00003: loss improved from 7.50084 to 4.94642, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 4/100
969/969 [==============================] - 678s 695ms/step - loss: 3.8672 - ssim: 0.9419 - accuracy: 0.7057

Epoch 00004: loss improved from 4.94642 to 4.18325, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 5/100
969/969 [==============================] - 677s 695ms/step - loss: 2.5206 - ssim: 0.9525 - accuracy: 0.7182

Epoch 00005: loss improved from 4.18325 to 3.14321, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 6/100
969/969 [==============================] - 679s 697ms/step - loss: 3.1717 - ssim: 0.9521 - accuracy: 0.7171

Epoch 00006: loss did not improve from 3.14321
Epoch 7/100
969/969 [==============================] - 681s 697ms/step - loss: 3.5549 - ssim: 0.9525 - accuracy: 0.7181

Epoch 00007: loss did not improve from 3.14321
Epoch 8/100
969/969 [==============================] - 679s 697ms/step - loss: 1.9836 - ssim: 0.9554 - accuracy: 0.7245

Epoch 00008: loss improved from 3.14321 to 2.39766, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 9/100
969/969 [==============================] - 678s 696ms/step - loss: 3.4578 - ssim: 0.9569 - accuracy: 0.7228

Epoch 00009: loss did not improve from 2.39766
Epoch 10/100
969/969 [==============================] - 679s 696ms/step - loss: 1.5053 - ssim: 0.9663 - accuracy: 0.7255

Epoch 00010: loss did not improve from 2.39766
Epoch 11/100
969/969 [==============================] - 679s 696ms/step - loss: 2.3102 - ssim: 0.9605 - accuracy: 0.7208

Epoch 00011: loss did not improve from 2.39766
Epoch 12/100
969/969 [==============================] - 679s 696ms/step - loss: 3.4977 - ssim: 0.9530 - accuracy: 0.7208

Epoch 00012: loss did not improve from 2.39766
Epoch 13/100
969/969 [==============================] - 679s 696ms/step - loss: 5.0734 - ssim: 0.9437 - accuracy: 0.7172

Epoch 00013: loss did not improve from 2.39766
Epoch 14/100
969/969 [==============================] - 678s 695ms/step - loss: 1.6401 - ssim: 0.9652 - accuracy: 0.7192

Epoch 00014: loss improved from 2.39766 to 1.79334, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 15/100
969/969 [==============================] - 677s 695ms/step - loss: 2.7825 - ssim: 0.9588 - accuracy: 0.7166

Epoch 00015: loss did not improve from 1.79334
Epoch 16/100
969/969 [==============================] - 677s 694ms/step - loss: 3.0589 - ssim: 0.9530 - accuracy: 0.7242

Epoch 00016: loss did not improve from 1.79334
Epoch 17/100
969/969 [==============================] - 684s 701ms/step - loss: 2.3687 - ssim: 0.9659 - accuracy: 0.7251

Epoch 00017: loss did not improve from 1.79334
Epoch 18/100
969/969 [==============================] - 702s 719ms/step - loss: 1.2221 - ssim: 0.9707 - accuracy: 0.7293

Epoch 00018: loss improved from 1.79334 to 1.45419, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 19/100
969/969 [==============================] - 702s 720ms/step - loss: 1.6338 - ssim: 0.9662 - accuracy: 0.7225

Epoch 00019: loss did not improve from 1.45419
Epoch 20/100
969/969 [==============================] - 700s 718ms/step - loss: 1.7770 - ssim: 0.9608 - accuracy: 0.7311

Epoch 00020: loss improved from 1.45419 to 1.39385, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 21/100
969/969 [==============================] - 700s 719ms/step - loss: 1.1112 - ssim: 0.9656 - accuracy: 0.7246

Epoch 00021: loss did not improve from 1.39385
Epoch 22/100
969/969 [==============================] - 679s 697ms/step - loss: 0.9955 - ssim: 0.9658 - accuracy: 0.7250

Epoch 00022: loss improved from 1.39385 to 1.10481, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 23/100
969/969 [==============================] - 677s 695ms/step - loss: 2.0299 - ssim: 0.9603 - accuracy: 0.7202

Epoch 00023: loss did not improve from 1.10481
Epoch 24/100
969/969 [==============================] - 678s 695ms/step - loss: 1.1907 - ssim: 0.9736 - accuracy: 0.7293

Epoch 00024: loss did not improve from 1.10481
Epoch 25/100
969/969 [==============================] - 678s 695ms/step - loss: 0.8573 - ssim: 0.9784 - accuracy: 0.7321

Epoch 00025: loss improved from 1.10481 to 0.53482, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 26/100
969/969 [==============================] - 682s 700ms/step - loss: 1.8557 - ssim: 0.9634 - accuracy: 0.7272

Epoch 00026: loss did not improve from 0.53482
Epoch 27/100
969/969 [==============================] - 685s 703ms/step - loss: 0.8739 - ssim: 0.9756 - accuracy: 0.7302

Epoch 00027: loss did not improve from 0.53482
Epoch 28/100
969/969 [==============================] - 682s 700ms/step - loss: 1.5764 - ssim: 0.9690 - accuracy: 0.7275

Epoch 00028: loss did not improve from 0.53482
Epoch 29/100
969/969 [==============================] - 684s 702ms/step - loss: 0.7666 - ssim: 0.9769 - accuracy: 0.7306

Epoch 00029: loss did not improve from 0.53482
Epoch 30/100
969/969 [==============================] - 685s 703ms/step - loss: 0.8856 - ssim: 0.9760 - accuracy: 0.7314

Epoch 00030: loss did not improve from 0.53482
Epoch 31/100
969/969 [==============================] - 684s 702ms/step - loss: 1.1426 - ssim: 0.9720 - accuracy: 0.7307

Epoch 00031: loss did not improve from 0.53482
Epoch 32/100
969/969 [==============================] - 685s 703ms/step - loss: 1.2693 - ssim: 0.9665 - accuracy: 0.7292

Epoch 00032: loss did not improve from 0.53482
Epoch 33/100
969/969 [==============================] - 681s 699ms/step - loss: 2.0898 - ssim: 0.9568 - accuracy: 0.7300

Epoch 00033: loss did not improve from 0.53482
Epoch 34/100
969/969 [==============================] - 677s 695ms/step - loss: 1.1165 - ssim: 0.9766 - accuracy: 0.7322

Epoch 00034: loss did not improve from 0.53482
Epoch 35/100
969/969 [==============================] - 678s 695ms/step - loss: 0.7079 - ssim: 0.9819 - accuracy: 0.7263

Epoch 00035: loss did not improve from 0.53482
Epoch 36/100
969/969 [==============================] - 679s 696ms/step - loss: 0.6872 - ssim: 0.9831 - accuracy: 0.7322

Epoch 00036: loss did not improve from 0.53482
Epoch 37/100
969/969 [==============================] - 678s 695ms/step - loss: 0.5325 - ssim: 0.9745 - accuracy: 0.7258

Epoch 00037: loss did not improve from 0.53482
Epoch 38/100
969/969 [==============================] - 678s 695ms/step - loss: 0.7614 - ssim: 0.9773 - accuracy: 0.7352

Epoch 00038: loss did not improve from 0.53482
Epoch 39/100
969/969 [==============================] - 677s 694ms/step - loss: 0.3590 - ssim: 0.9861 - accuracy: 0.7210

Epoch 00039: loss improved from 0.53482 to 0.33833, saving model to ../model/model_8_3_mse_relu_v1.hdf5
Epoch 40/100
969/969 [==============================] - 677s 695ms/step - loss: 0.8627 - ssim: 0.9786 - accuracy: 0.7304

Epoch 00040: loss did not improve from 0.33833
Epoch 41/100
969/969 [==============================] - 680s 697ms/step - loss: 1.4624 - ssim: 0.9667 - accuracy: 0.7251

Epoch 00041: loss did not improve from 0.33833
Epoch 42/100
313/969 [========>.....................] - ETA: 7:33 - loss: 0.2189 - ssim: 0.9875 - accuracy: 0.7424Traceback (most recent call last):
  File "/home/allen/Project/deep-learning/PlanDoseQA/by3d/train3d.py", line 47, in <module>
    use_multiprocessing=False, workers=10, max_queue_size=10, callbacks=[checkpoint])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 855, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2943, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
KeyboardInterrupt
2022-04-25 17:20:08.094452: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]
terminate called without an active exception

Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
