[Rtk-users] [EXTERNAL] Clearing cuda variables in python
Simon Rit
simon.rit at creatis.insa-lyon.fr
Mon Jun 19 11:41:04 CEST 2023
This option has not been implemented, no. You can use float64 in some cases
but we never worked with float16.
Simon
On Mon, Jun 19, 2023 at 11:37 AM Rahman, Obaid <rahmano at ornl.gov> wrote:
> Got it. Thank you.
> When I increase the voxel size to twice and image dimension to half, it
> runs without any memory error.
>
> I was wondering if I’d be able to work with float16 images and
> projections, since the default datatype is float32 for the filters I am
> using.
>
> On Jun 19, 2023, at 1:07 AM, Simon Rit <simon.rit at creatis.insa-lyon.fr>
> wrote:
>
> Hi,
> Yes, RTK only uses one GPU. We haven't implemented multi-GPU capability.
> Any contribution in that direction is welcome!
> If you really have 32 GB available for GPU0, then I don't understand what
> is the issue but you should be able to find out.
> Simon
>
> On Fri, Jun 16, 2023 at 6:32 PM Rahman, Obaid <rahmano at ornl.gov> wrote:
>
>> Thank you, Simon and Nils, for the suggestions.
>>
>> This is what I am doing:
>> a. I read a (Xray CT) projection data (~1.5 GB)
>> b. Then perform FDK reconstruction (~10 GB)
>> c. Then forward project that reconstruction
>>
>> I am using the cuda version of itk-rtk. Steps a and b work fine and the
>> reconstruction looks as expected.
>> Then these are the lines of code I use for forward projection:
>>
>> ForwardProj = rtk.CudaForwardProjectionImageFilter[CudaImageType].New()
>> ForwardProj.SetGeometry( geometry )
>> ForwardProj.SetInput(0, constantImageSource.GetOutput())
>> ForwardProj.SetInput(1, recon_image.GetOutput())
>> ForwardProj.Update()
>>
>> I get this error:
>> Traceback (most recent call last):
>> File "projection_test.py", line 223, in <module>
>> ForwardProj.Update()
>> RuntimeError: /work/src/rtkCudaUtilities.cu:115:
>> ITK ERROR: CUDA ERROR: out of memory
>>
>> I also try to delete the variables I don’t need using (del variable),
>> but I still get that error.
>> The reconstruction is supposed to be ~10GB, projection is supposed to be
>> 1.5 GB. GPU0 memory is 32 GB.
>>
>> I can see that before I get the error, GPU0 memory reaches about its
>> capacity.
>> The weird thing is that GPU1, GPU2, GPU3 that have the same memory
>> capacity are basically unused (~4MB each).
>> Is the forward projection not performed over all 4 GPUs?
>>
>> To try to force the code to use all 4 GPUs, I have also included these
>> lines in the code:
>> os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
>> os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2,3" # specify which GPU(s) to
>> be used
>>
>> But I still get that error, and only GPU0 gets used.
>>
>>
>> Best,
>> Obaid
>>
>> On Jun 16, 2023, at 8:58 AM, krah <nils.krah at creatis.insa-lyon.fr> wrote:
>>
>> Just to add to Simon's answer:
>> Objects are deleted by python's garbage collection mechanism once there
>> are no references left to the object. (keyword: reference counting).
>> The
>> del variable
>> statement Simon mention deletes the reference variable, but the memory
>> is freed only when python runs a garbage collection. That might immediately
>> afterwards, or slightly later (scheduled). Usually, you do not need to
>> worry about that and deleting the reference as Simon said is sufficient. If
>> for some reason you need to be absolutely sure that the object is garbage
>> collected immediately, you can trigger a garbage collection by
>>
>> import gc
>> gc.collect()
>>
>> Again, that should not be necessary, but it is useful to bear in mind how
>> the memory is actually handled in python.
>>
>> Cheers,
>> Nils
>>
>> On Jun 16 2023, at 9:10 am, Simon Rit <simon.rit at creatis.insa-lyon.fr>
>> wrote:
>>
>>> Hi,
>>> If you have a Python variable holding memory, you can simply call
>>> del variable_name
>>> If you want to clear the GPU memory for an image, you can access the CPU
>>> buffer pointer to have it moved to the computer RAM. Examples below.
>>> Simon
>>>
>>> >>> import os
>>> >>> os.system('nvidia-smi')
>>> Fri Jun 16 08:49:38 2023
>>>
>>> +-----------------------------------------------------------------------------+
>>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version:
>>> 11.6 |
>>>
>>> |-------------------------------+----------------------+----------------------+
>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>> Uncorr. ECC |
>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>> Compute M. |
>>> | | |
>>> MIG M. |
>>>
>>> |===============================+======================+======================|
>>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>>> N/A |
>>> | N/A 52C P8 N/A / N/A | 0MiB / 4096MiB | 0%
>>> Default |
>>> | | |
>>> N/A |
>>>
>>> +-------------------------------+----------------------+----------------------+
>>>
>>>
>>>
>>> +-----------------------------------------------------------------------------+
>>> | Processes:
>>> |
>>> | GPU GI CI PID Type Process name GPU
>>> Memory |
>>> | ID ID Usage
>>> |
>>>
>>> |=============================================================================|
>>> | No running processes found
>>> |
>>>
>>> +-----------------------------------------------------------------------------+
>>> 0
>>> >>> import itk
>>> >>> from itk import RTK as rtk
>>> >>> img=itk.CudaImage[itk.F, 3].New()
>>> >>> img.SetRegions([500]*3)
>>> >>> img.Allocate()
>>> >>> os.system('nvidia-smi')
>>> Fri Jun 16 08:51:28 2023
>>>
>>> +-----------------------------------------------------------------------------+
>>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version:
>>> 11.6 |
>>>
>>> |-------------------------------+----------------------+----------------------+
>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>> Uncorr. ECC |
>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>> Compute M. |
>>> | | |
>>> MIG M. |
>>>
>>> |===============================+======================+======================|
>>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>>> N/A |
>>> | N/A 58C P8 N/A / N/A | 45MiB / 4096MiB | 0%
>>> Default |
>>> | | |
>>> N/A |
>>>
>>> +-------------------------------+----------------------+----------------------+
>>>
>>>
>>>
>>> +-----------------------------------------------------------------------------+
>>> | Processes:
>>> |
>>> | GPU GI CI PID Type Process name GPU
>>> Memory |
>>> | ID ID Usage
>>> |
>>>
>>> |=============================================================================|
>>> | 0 N/A N/A 10890 C python
>>> 43MiB |
>>>
>>> +-----------------------------------------------------------------------------+
>>> 0
>>> >>> img.GetCudaDataManager().GetGPUBufferPointer()
>>> <Swig Object of type 'void *' at 0x7f20d3327f60>
>>> >>> os.system('nvidia-smi')
>>> Fri Jun 16 08:52:05 2023
>>>
>>> +-----------------------------------------------------------------------------+
>>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version:
>>> 11.6 |
>>>
>>> |-------------------------------+----------------------+----------------------+
>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>> Uncorr. ECC |
>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>> Compute M. |
>>> | | |
>>> MIG M. |
>>>
>>> |===============================+======================+======================|
>>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>>> N/A |
>>> | N/A 54C P0 N/A / N/A | 525MiB / 4096MiB | 0%
>>> Default |
>>> | | |
>>> N/A |
>>>
>>> +-------------------------------+----------------------+----------------------+
>>>
>>>
>>>
>>> +-----------------------------------------------------------------------------+
>>> | Processes:
>>> |
>>> | GPU GI CI PID Type Process name GPU
>>> Memory |
>>> | ID ID Usage
>>> |
>>>
>>> |=============================================================================|
>>> | 0 N/A N/A 10890 C python
>>> 523MiB |
>>>
>>> +-----------------------------------------------------------------------------+
>>> 0
>>> >>> del img
>>> >>> os.system('nvidia-smi')
>>> Fri Jun 16 08:52:21 2023
>>>
>>> +-----------------------------------------------------------------------------+
>>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version:
>>> 11.6 |
>>>
>>> |-------------------------------+----------------------+----------------------+
>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>> Uncorr. ECC |
>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>> Compute M. |
>>> | | |
>>> MIG M. |
>>>
>>> |===============================+======================+======================|
>>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>>> N/A |
>>> | N/A 56C P0 N/A / N/A | 2MiB / 4096MiB | 0%
>>> Default |
>>> | | |
>>> N/A |
>>>
>>> +-------------------------------+----------------------+----------------------+
>>>
>>>
>>>
>>> +-----------------------------------------------------------------------------+
>>> | Processes:
>>> |
>>> | GPU GI CI PID Type Process name GPU
>>> Memory |
>>> | ID ID Usage
>>> |
>>>
>>> |=============================================================================|
>>> | No running processes found
>>> |
>>>
>>> +-----------------------------------------------------------------------------+
>>> 0
>>> >>> img=itk.CudaImage[itk.F, 3].New()
>>> >>> img.SetRegions([500]*3)
>>> >>> img.GetCudaDataManager().GetGPUBufferPointer()
>>> <Swig Object of type 'void *' at 0x7f20b2c30870>
>>> >>> os.system('nvidia-smi')
>>> Fri Jun 16 08:54:10 2023
>>>
>>> +-----------------------------------------------------------------------------+
>>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version:
>>> 11.6 |
>>>
>>> |-------------------------------+----------------------+----------------------+
>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>> Uncorr. ECC |
>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>> Compute M. |
>>> | | |
>>> MIG M. |
>>>
>>> |===============================+======================+======================|
>>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>>> N/A |
>>> | N/A 58C P0 N/A / N/A | 525MiB / 4096MiB | 0%
>>> Default |
>>> | | |
>>> N/A |
>>>
>>> +-------------------------------+----------------------+----------------------+
>>>
>>>
>>>
>>> +-----------------------------------------------------------------------------+
>>> | Processes:
>>> |
>>> | GPU GI CI PID Type Process name GPU
>>> Memory |
>>> | ID ID Usage
>>> |
>>>
>>> |=============================================================================|
>>> | 0 N/A N/A 10890 C python
>>> 523MiB |
>>>
>>> +-----------------------------------------------------------------------------+
>>> 0
>>> >>> img.GetCudaDataManager().GetCPUBufferPointer()
>>> >>> os.system('nvidia-smi')
>>> Fri Jun 16 08:54:20 2023
>>>
>>> +-----------------------------------------------------------------------------+
>>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version:
>>> 11.6 |
>>>
>>> |-------------------------------+----------------------+----------------------+
>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>> Uncorr. ECC |
>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>> Compute M. |
>>> | | |
>>> MIG M. |
>>>
>>> |===============================+======================+======================|
>>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>>> N/A |
>>> | N/A 58C P0 N/A / N/A | 47MiB / 4096MiB | 0%
>>> Default |
>>> | | |
>>> N/A |
>>>
>>> +-------------------------------+----------------------+----------------------+
>>>
>>>
>>>
>>> +-----------------------------------------------------------------------------+
>>> | Processes:
>>> |
>>> | GPU GI CI PID Type Process name GPU
>>> Memory |
>>> | ID ID Usage
>>> |
>>>
>>> |=============================================================================|
>>> | 0 N/A N/A 10890 C python
>>> 45MiB |
>>>
>>> +-----------------------------------------------------------------------------+
>>> 0
>>>
>>> On Thu, Jun 15, 2023 at 5:33 PM Rahman, Obaid <rahmano at ornl.gov> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using ink-rtk-cuda116 with *python*.
>>>> I have too many cuda variables (images and filters).
>>>> I would like to clear some of these variables.
>>>>
>>>> I am getting the following error:
>>>> ITK ERROR: CUDA ERROR: out of memory
>>>>
>>>> Does anyone know how to clear Cuda variables in python?
>>>> Thanks.
>>>>
>>>> Best,
>>>> Obaidullah Rahman
>>>> Oak Ridge National Laboratory, TN, United States
>>>> _______________________________________________
>>>> Rtk-users mailing list
>>>> rtk-users at openrtk.org
>>>> https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users
>>>> <https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=>
>>>>
>>> _______________________________________________
>>> Rtk-users mailing list
>>> rtk-users at openrtk.org
>>> https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users
>>> <https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwQFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.creatis.insa-lyon.fr/pipermail/rtk-users/attachments/20230619/af750750/attachment-0001.htm>
More information about the Rtk-users
mailing list