[Rtk-users] [EXTERNAL] Clearing cuda variables in python
Rahman, Obaid
rahmano at ornl.gov
Mon Jun 19 11:37:03 CEST 2023
Got it. Thank you.
When I increase the voxel size to twice and image dimension to half, it runs without any memory error.
I was wondering if I’d be able to work with float16 images and projections, since the default datatype is float32 for the filters I am using.
On Jun 19, 2023, at 1:07 AM, Simon Rit <simon.rit at creatis.insa-lyon.fr> wrote:
Hi,
Yes, RTK only uses one GPU. We haven't implemented multi-GPU capability. Any contribution in that direction is welcome!
If you really have 32 GB available for GPU0, then I don't understand what is the issue but you should be able to find out.
Simon
On Fri, Jun 16, 2023 at 6:32 PM Rahman, Obaid <rahmano at ornl.gov<mailto:rahmano at ornl.gov>> wrote:
Thank you, Simon and Nils, for the suggestions.
This is what I am doing:
a. I read a (Xray CT) projection data (~1.5 GB)
b. Then perform FDK reconstruction (~10 GB)
c. Then forward project that reconstruction
I am using the cuda version of itk-rtk. Steps a and b work fine and the reconstruction looks as expected.
Then these are the lines of code I use for forward projection:
ForwardProj = rtk.CudaForwardProjectionImageFilter[CudaImageType].New()
ForwardProj.SetGeometry( geometry )
ForwardProj.SetInput(0, constantImageSource.GetOutput())
ForwardProj.SetInput(1, recon_image.GetOutput())
ForwardProj.Update()
I get this error:
Traceback (most recent call last):
File "projection_test.py", line 223, in <module>
ForwardProj.Update()
RuntimeError: /work/src/rtkCudaUtilities.cu:115:
ITK ERROR: CUDA ERROR: out of memory
I also try to delete the variables I don’t need using (del variable), but I still get that error.
The reconstruction is supposed to be ~10GB, projection is supposed to be 1.5 GB. GPU0 memory is 32 GB.
I can see that before I get the error, GPU0 memory reaches about its capacity.
The weird thing is that GPU1, GPU2, GPU3 that have the same memory capacity are basically unused (~4MB each).
Is the forward projection not performed over all 4 GPUs?
To try to force the code to use all 4 GPUs, I have also included these lines in the code:
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2,3" # specify which GPU(s) to be used
But I still get that error, and only GPU0 gets used.
Best,
Obaid
On Jun 16, 2023, at 8:58 AM, krah <nils.krah at creatis.insa-lyon.fr<mailto:nils.krah at creatis.insa-lyon.fr>> wrote:
Just to add to Simon's answer:
Objects are deleted by python's garbage collection mechanism once there are no references left to the object. (keyword: reference counting).
The
del variable
statement Simon mention deletes the reference variable, but the memory is freed only when python runs a garbage collection. That might immediately afterwards, or slightly later (scheduled). Usually, you do not need to worry about that and deleting the reference as Simon said is sufficient. If for some reason you need to be absolutely sure that the object is garbage collected immediately, you can trigger a garbage collection by
import gc
gc.collect()
Again, that should not be necessary, but it is useful to bear in mind how the memory is actually handled in python.
Cheers,
Nils
On Jun 16 2023, at 9:10 am, Simon Rit <simon.rit at creatis.insa-lyon.fr<mailto:simon.rit at creatis.insa-lyon.fr>> wrote:
Hi,
If you have a Python variable holding memory, you can simply call
del variable_name
If you want to clear the GPU memory for an image, you can access the CPU buffer pointer to have it moved to the computer RAM. Examples below.
Simon
>>> import os
>>> os.system('nvidia-smi')
Fri Jun 16 08:49:38 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A |
| N/A 52C P8 N/A / N/A | 0MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
0
>>> import itk
>>> from itk import RTK as rtk
>>> img=itk.CudaImage[itk.F, 3].New()
>>> img.SetRegions([500]*3)
>>> img.Allocate()
>>> os.system('nvidia-smi')
Fri Jun 16 08:51:28 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A |
| N/A 58C P8 N/A / N/A | 45MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 10890 C python 43MiB |
+-----------------------------------------------------------------------------+
0
>>> img.GetCudaDataManager().GetGPUBufferPointer()
<Swig Object of type 'void *' at 0x7f20d3327f60>
>>> os.system('nvidia-smi')
Fri Jun 16 08:52:05 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A |
| N/A 54C P0 N/A / N/A | 525MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 10890 C python 523MiB |
+-----------------------------------------------------------------------------+
0
>>> del img
>>> os.system('nvidia-smi')
Fri Jun 16 08:52:21 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A |
| N/A 56C P0 N/A / N/A | 2MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
0
>>> img=itk.CudaImage[itk.F, 3].New()
>>> img.SetRegions([500]*3)
>>> img.GetCudaDataManager().GetGPUBufferPointer()
<Swig Object of type 'void *' at 0x7f20b2c30870>
>>> os.system('nvidia-smi')
Fri Jun 16 08:54:10 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A |
| N/A 58C P0 N/A / N/A | 525MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 10890 C python 523MiB |
+-----------------------------------------------------------------------------+
0
>>> img.GetCudaDataManager().GetCPUBufferPointer()
>>> os.system('nvidia-smi')
Fri Jun 16 08:54:20 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P2000 Off | 00000000:01:00.0 Off | N/A |
| N/A 58C P0 N/A / N/A | 47MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 10890 C python 45MiB |
+-----------------------------------------------------------------------------+
0
On Thu, Jun 15, 2023 at 5:33 PM Rahman, Obaid <rahmano at ornl.gov<mailto:rahmano at ornl.gov>> wrote:
Hi,
I am using ink-rtk-cuda116 with python.
I have too many cuda variables (images and filters).
I would like to clear some of these variables.
I am getting the following error:
ITK ERROR: CUDA ERROR: out of memory
Does anyone know how to clear Cuda variables in python?
Thanks.
Best,
Obaidullah Rahman
Oak Ridge National Laboratory, TN, United States
_______________________________________________
Rtk-users mailing list
rtk-users at openrtk.org<mailto:rtk-users at openrtk.org>
https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users<https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=>
_______________________________________________
Rtk-users mailing list
rtk-users at openrtk.org<mailto:rtk-users at openrtk.org>
https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users<https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwQFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.creatis.insa-lyon.fr/pipermail/rtk-users/attachments/20230619/380d50b9/attachment-0001.htm>
More information about the Rtk-users
mailing list