logo
Loading...

Colab GPU: Resource exhausted - Cupoy

加了 allow_growth 都未能跑完, 請問如何解決呢?config = tf.ConfigP...

Colab

Colab GPU: Resource exhausted

2020/04/12 10:53 PM
電腦視覺深度學習論壇
WP
觀看數:0
回答數:1
收藏數:0
Colab

加了 allow_growth 都未能跑完, 請問如何解決呢?

config = tf.ConfigProto()
config.gpu_options.allow_growth = True


Epoch 50/50

5/5 [==============================] - 7s 1s/step - loss: 72.5174 - val_loss: 70.2399

Unfreeze all of the layers.

Train on 90 samples, val on 10 samples, with batch size 16.

Epoch 51/100

---------------------------------------------------------------------------

ResourceExhaustedError                    Traceback (most recent call last)

in ()

    70         epochs=100,

    71         initial_epoch=50,

---> 72         callbacks=[logging, checkpoint, reduce_lr, early_stopping])

    73     model.save_weights(log_dir + 'trained_weights_final.h5')


6 frames

/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in __call__(self, *args, **kwargs)

  1470         ret = tf_session.TF_SessionRunCallable(self._session._session,

  1471                                                self._handle, args,

-> 1472                                                run_metadata_ptr)

  1473         if run_metadata:

  1474           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)


ResourceExhaustedError: 2 root error(s) found.

 (0) Resource exhausted: OOM when allocating tensor with shape[16,105,105,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

[[{{node zero_padding2d_3/Pad}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


 (1) Resource exhausted: OOM when allocating tensor with shape[16,105,105,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

[[{{node zero_padding2d_3/Pad}}]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


[[loss_1/add_74/_5299]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


0 successful operations.

0 derived errors ignored.