Help with hardware (CUDA)

I’m trying to setup HQPlayer on a dedicated machine to squeeze as much as possible from it. But I think maybe cuda isn’t working as should? I have installed all the various cuda drivers but it seems HQPlayer uses very little memory from the GPU. While playing a track this is the output from the GPU using the command nvidia-smi

image

Looking at the process for hqplayerd its only using 145MiB? should this not be a lot higher?

Some other info I have found using the nvcc --version command

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

My processor is a AMD Ryzen 3 2200G, 16gig ram. During playback the speeds of the cores seem to rise around 3.6Ghz (2 examples below)

cpu MHz         : 3425.257
cpu MHz         : 3421.047
cpu MHz         : 3442.948
cpu MHz         : 3471.218
cpu MHz         : 3693.204
cpu MHz         : 3693.213
cpu MHz         : 2460.553
cpu MHz         : 2798.226

And finally a screenshot of roon’s signal path:
image

So… is cuda working? what can I do to improve the performance?

Just for info I followed this guide for installing cuda.

Looks like it is working as it should, hqplayerd is shown as “C” (compute) process as opposed to “G” (graphics). 145 MB is quite a lot of memory already. And poly-sinc-short-mp-2s among the lightest to process of all the filters. If you try with something like poly-sinc-xtr-lp (non-2s), or poly-sinc-short-mp instead, you’ll likely get higher figures.

I’m a little curious though why do you have graphical desktop on a dedicated machine with HQPlayer Embedded?

That is also normal, or actually quite low, because typical figures can easily go to 4 - 5 GHz range with ASDM7EC. But since your output is only DSD128 you get along with lower figure.

This got installed during the graphics drivers installation. I was going to do a full reinstall once I got the proper guides to follow to get everything running smoothly.

The figures for this option seems to be very similar:

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       891      G   /usr/lib/xorg/Xorg                  8MiB |
|    0   N/A  N/A      1014      G   /usr/bin/gnome-shell                2MiB |
|    0   N/A  N/A      4684      C   hqplayerd                         117MiB |
+-----------------------------------------------------------------------------+
cpu MHz         : 3266.789
cpu MHz         : 3297.548
cpu MHz         : 2853.765
cpu MHz         : 2855.525

But does stutter a bit.

First, get started by installing minimal Ubuntu Server 20.04 LTS (with just SSH server, nothing else). Then I would recommend to just add the CUDA repository and pull in for example “cuda-drivers” meta-package. That will also pull in lot of unnecessary stuff, but will keep things up to date and doesn’t force graphical desktop. Alternatively, more minimal by installing “nvidia-dkms-470” or “nvidia-driver-470”, but you’d need to watch out for major version updates manually.

How’s your GPU load? Which GPU is it? Name doesn’t fit in the screenshot of nvidia-smi.

Yeah I think I will go for a clean install again.

It is an RTX2060 & I am using ubuntu20.04

Gonna have to get the thinking cap on :slight_smile:
libhsakmt.so.1: cannot open shared object file:
This error has came up after a fresh install of ubuntu server 20.04 and following the guide for hqplayerd

All dependencies are in after “sudo apt install -f”, including the ROCm stuff?

AMD is changing their packaging stuff on each release, but they don’t always seem to automatically include their library path in ld.so’s search path, so manually doing this will help:

sudo echo "/opt/rocm/lib" >/etc/ld.so.conf.d/rocm.conf
sudo ldconfig

This works as long as they establish symbolic link (through alternatives) from the version specific directory to /opt/rocm, which they have been doing so far.

I think I may have got sorted. I have done a full fresh install. I still managed to run into a couple of hiccups but nothing too brain taxing.
I tried to copy all the commands and steps as I took them for future ref.

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub                                                                           
 sudo bash -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'                                                       
 sudo bash -c 'echo "deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda_learn.list'                                     
 sudo apt update                                                                                                                                                                                 
 sudo apt install cuda-drivers                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                 
 wget https://www.sonarnerd.net/src/focal/libgmpris_2.2.1-8_amd64.deb                                                                                                                            
 sudo dpkg -i libgmpris_2.2.1-8_amd64.deb                                                                                                                                                        
                                                                                                                                                                                                 
 sudo usermod -a -G video $LOGNAME                                                                                                                                                               
 echo 'ADD_EXTRA_GROUPS=1' | sudo tee -a /etc/adduser.conf                                                                                                                                       
 echo 'EXTRA_GROUPS=video' | sudo tee -a /etc/adduser.conf                                                                                                                                       
 echo 'EXTRA_GROUPS=render' | sudo tee -a /etc/adduser.conf                                                                                                                                      
 sudo apt update                                                                                                                                                                                 
 sudo apt dist-upgrade                                                                                                                                                                           
 sudo apt install libnuma-dev                                                                                                                                                                    
 sudo reboot                                                                                                                                                                                     
                                                                                                                                                                                                 
 sudo apt install wget gnupg2                                                                                                                                                                    
 wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add -                                                                                                                     
 echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/debian/ ubuntu main' | sudo tee /etc/apt/sources.list.d/rocm.list                                                                       
 sudo apt update                                                                                                                                                                                 
                                                                                                                                                                                                 
 wget https://www.signalyst.eu/bins/hqplayerd/focal/hqplayerd_4.25.2-86amd_amd64.deb                                                                                                             
 sudo dpkg -i hqplayerd_4.25.2-86amd_amd64.deb                                                                                                                                                   
 sudo apt install -f                                                                                                                                                                             
 sudo echo "/opt/rocm/lib" >/etc/ld.so.conf.d/rocm.conf                                                                                                                                          
 sudo ldconfig                                                                                                                                                                                   

After the cuda-drivers install xorg was still installed. This command prevents the server from going to sleep
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

This process did still install xserver when installing the cuda drivers, I found a command afterwards which may fix this sudo apt-get install --no-install-recommends cuda-drivers

Also for some odd reason the soundcard wouldn’t show up on alsa? To fix this I ran these commands:

sudo apt-get --purge remove linux-sound-base alsa-base alsa-utils
sudo apt-get install linux-sound-base alsa-base alsa-utils

And a look at what nvidia-smi looks like with a track playing through roon → HQPlayer:
image
And my signal path looks like this:
image
All playing with no glitches.
But I am still not happy lol. I’m not completely convinced I am using the full potential of cuda?

I will be doing another fresh install using the command sudo apt-get install --no-install-recommends cuda-drivers to see if it installs without all the xorg guff. But will also want to absolutely happy i’m using the full potential of the GFX card.

My mission continues.

Check that you don’t accidentally have pulseaudio installed, it would interfere…

Good find @jussi_laako just checked and yes.
image
I think it may have been installed with all the other xorg stuff.

Well I found a nice wee tool called nvtop which seems to show the usage of my GPU.
image
It would seem playing a 192/24khz file with the same filters shown a few posts back the gpu is only running at around 35%.
So… Is it possible to increase this usage. Eg. offload more cpu usage to cuda?
Swapped about with the filters and can bring the gpu up to 100% so its working perfect!
I have noticed that some filters seem to use more cpu than gpu so time for a bigger cpu now that I am certain I am getting the full potential from my gpu.
Thanks @jussi_laako :+1:

I just got RTX A4000 today, but hqplayer isn’t using it for some reason.

❯ nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-)

❯ nvidia-smi
Tue Mar 7 21:45:37 2023
±----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A4000 Off | 00000000:01:00.0 Off | Off |
| 41% 37C P8 13W / 140W | 4MiB / 16376MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3280 G …hell-43.2/bin/gnome-shell 3MiB |
±----------------------------------------------------------------------------+

Right now I’m using integrated graphics for monitor to check if that would help, but nothing. When I was using RTX A4000 for monitor, I saw for example firefox in that list as well.

In my /etc/hqplayer/hqplayerd.xml I have:
<engine … cuda=“1” …>

To me it looks like I have all the necessary drivers installed and everything works just fine otherwise. HQPlayer just isn’t using it. My personal guess is that it just can’t find the card for some reason.

From where does HQPlayer try to find the card as nvidia-smi can show the card just fine?

If you have latest HQPlayer, then the driver is too old. In the nvidia-smi it shows “CUDA Version: 11.8” while HQPlayer requires driver that supports latest CUDA which is 12.1. This means driver >= 530

Easiest way to handle this is to add the Nvidia’s CUDA repository according to instructions here.

Then just install package called “nvidia-headless-530” this should pull in drivers needed for headless server operation.

Note that they don’t have a generic headless package, so once HQPlayer is updated with new CUDA version again, then you need to manually look up the latest headless package and move over to a new version.

Thanks for the pointer. I’m indeed using 4.34. I didn’t know that it wouldn’t be backwards compatible with older drivers.

Hmm, this just became a problem to me as my operating system (NixOS) doesn’t have CUDA 12 available yet and installing it by hand would be… tricky.

Nvidia seems to have a tight pairing between the CUDA stack and their driver. Generally applications built with older CUDA SDK work with newer drivers, but not vice versa.

If the driver is not new enough the CUDA stack doesn’t get initialized and it just reports that there are no GPUs available. In such case HQPlayer just falls back to the regular CPU code.

I now managed to get to 12.0 using unstable channels for NixOS, but I still got the same results.

±----------------------------------------------------------------------------+
| NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A4000 Off | 00000000:01:00.0 Off | Off |
| 41% 34C P8 12W / 140W | 4MiB / 16376MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2082 G …hell-43.3/bin/gnome-shell 3MiB |
±----------------------------------------------------------------------------+

The HQPlayer build you are using is one with CUDA support?
IOW, Ubuntu build, Debian build or regular Fedora build (not the SSE4.2 one)?

I think the driver needs to be CUDA 12.1 compatible, >= 530.

Version used
https://www.signalyst.eu/bins/hqplayerd/fc36/hqplayerd-4.34.0-100sse42.fc36.x86_64.rpm

Ok, I’ll try to work around CUDA 12.1. It may be difficult as it’s not available in the unstable channel directly.

This “sse” was a good pointer. Manually switched to non-sse one and everything is working! What does the “sse” stand for? I didn’t find anything with quick googling.

EDIT: I can now listen to gauss-xla on PCM192->DSD1024. I’m a super happy camper!

1 Like