Forward compatibility on HQPe config files

[ipeverywhere]

tagging @jussi_laako

Jussi, What’s your policy on forward compatibility of the files in /etc/hqplayer for embedded?

Reason I ask… I boot embedded from a USB stick. I’ve created a script to copy files and users from the “active” system to a secondary USB stick for the purpose of upgrading HQP embedded. I would like to share this script but not if its going to eventually break someone’s install because a change was made to the format of these files.

Other questions for you if you could please answer:

1. Is there a file or other way on the filesystem to determine version? I’ve only found the version is part of the /about web server which requires a running system to find the version.

2. For the files in /etc/hqplayer… if you published the schema for the various files I could, possibly, validate the old config against the new schema and bomb out of my script before putting incompatible config in place.

Thank you sir and have an excellent day.

[jussi_laako (Jussi Laako (Signalyst))]

How much different are these from the already existing backup/restore scripts? (4.34.1 have still some migration issues to be fixed in next release)

The format itself is supposed to be forward compatible. Also backward compatible, but if you go backwards, and the new configuration for example refers to a filter that doesn’t exist in the old version, then HQPlayer will pick default filter until the config is updated.

For some version changes, HQPlayer has built-in conversion code from old config to a new one. For example when HQPlayer v3 was upgraded to HQPlayer v4, the configuration files changed and there’s a conversion for this case.

I don’t have any schema written for the XML’s.

[ipeverywhere]

@jussi_laako

How much different are these from the already existing backup/restore scripts?

Do note I boot from USB stick and my HPQe machine has no drives.

My process is basically:

1. Etch new embedded image on USB stick
2. Plug USB stick into active HQPe machine
3. Login (ssh) and run script
4. Reboot machine
5. Pull old USB stick out of machine while reboot occurs
6. Machine boots new HQPe, at correct address, with all config in place including my ssh user(s).

My script at step 3 (written in bash) handles copying over users and their directories (needed for ssh), groups, the HQPe config files including license and hqplayer web password, and network config (I use a static address).

I’m happy to share privately but my hacky “code” I don’t feel comfortable publishing here.

Anyway, this is faster than the current backup / restore unless you’ve got some backup / restore which handles users and network and can be exploded onto a USB stick so first boot is “ready to go”.

[jussi_laako (Jussi Laako (Signalyst))]

You can also replace the existing backup/restore scripts if you like to access it from HQPlayer web interface. Since HQPlayer Embedded is designed with hardware manufacturers in mind, there’s some extra flexibility.

Now in latest version, for example SMB mounts move over with the backups. But I need to fix the file ownership issues, especially when restoring from previous versions where the files were owned by different user id.

[ipeverywhere]

@jussi_laako

like to access it from HQPlayer web interface

oh, that maybe a good idea. Although, I prefer the command line.

ok, I’ll share my ugly “code” for moving users with you.

Edit:
Code removed so you don't use it and break things.

[ipeverywhere]

ohhh… and this question…

Is there a file or other way on the filesystem to determine version? I’ve only found the version is part of the /about web server which requires a running system to find the version.

[jussi_laako (Jussi Laako (Signalyst))]

@ipeverywhere

	cp /etc/passwd $TARGET/etc/passwd
	cp /etc/shadow $TARGET/etc/shadow
	cp /etc/group $TARGET/etc/group

This part breaks the OS for example between 4.34.0 and 4.34.1. So it would be better to do it with sed or similar to selectively move over only the created user id’s. For example on the 4.34.1 restore script I only pick the smb mount entry to move over to target fstab.

@ipeverywhere

Is there a file or other way on the filesystem to determine version? I’ve only found the version is part of the /about web server which requires a running system to find the version.

Not at the moment. I can think about how to add some file somewhere to state this. I would prefer it to be automatic from the build system, so I need to do some testing to figure this out.

[ipeverywhere]

@jussi_laako

So it would be better to do it with sed or similar

That’s pretty straightforward and a good suggestion. I wasn’t happy with blatantly overwriting these files anyway. Thanks

I have only tested this with 4.33 → 4.34

Have not had time to upgrade to 34.1 yet. But… its now on my list and I’ll make updating the script part of that work.

I don’t use SMB mounts as I’m Roon → HQP always. However, you bring up a good point that I should copy over SMB mounts so others can use this script. I need to set that up to test. I also don’t grab things like convolution filters because I don’t use that either. <sigh> Now this just became a bigger project oh well, it’s forcing me to dust off my worthless bash skills.

[ipeverywhere]

@jussi_laako

do it with sed or similar to selectively move over only the created user id’s.

OK, I updated this. I’m even sorting the move by oid so that the users go back in order. I tested this using 4.34 → 4.34.2

I’ll wait for the next release to verify everything is good before I share more “code”.

@jussi_laako

1. About says 4.34.0, do you normally not update the minor versions in the web (or did I goof)? Have you had a chance to think about exposing the version via a file please? Only way I know to get this from shell is:

CURV=`wget -qO- http://localhost/about |grep -A1 "<h3>Version</h3>"|grep -v Version`

2. In another thread a user found the web interface on 8080 and you said it moved ports? Mine still living on 80. Are you planning to move it?

Thread for reference:
https://community.roonlabs.com/t/hqplayer-embedded-thread/160210/1681?u=ipeverywhere

Thanks as always!

[jussi_laako (Jussi Laako (Signalyst))]

@ipeverywhere

About says 4.34.0, do you normally not update the minor versions in the web (or did I goof)?

@ipeverywhere

In another thread a user found the web interface on 8080 and you said it moved ports? Mine still living on 80. Are you planning to move it?

Yes, I have updated the version on web interface.

Sounds like you are still on 4.34.0? Yes I did move, because 80 is privileged port, and hqplayerd running as “hqplayer” user doesn’t have privileges to bind to such port.

@ipeverywhere

Have you had a chance to think about exposing the version via a file please?

I’ll do that right away now. The build process should be able to write it out.

[jussi_laako (Jussi Laako (Signalyst))]

@ipeverywhere

Have you had a chance to think about exposing the version via a file please?

Done, next release will have /var/lib/hqplayer/web/version.txt that has nothing else except the version number. You can also reach this through the internal web interface using for example http://hqplayer.local:8088/files/version.txt
It will return just the plain file, no HTML stuff.

[ipeverywhere]

@jussi_laako

Sounds like you are still on 4.34.0?

well… hmmm… Nope. But this is an issue:

-       <web port="8088" css="default.css"/>
+       <web css="default.css" port="80"/>

So, I’m eating your change because I blindly copied the old hqp files. I’m not sure there is a great way around this from the terminal.

Obviously, for any kind of upgrade to work I’m need to be on the new version. Maybe I can copy the config over and initiate the loading of the config using local tools (curl) to interact with the web interface on first boot?

For now, since I’ve kind of broken things, I’m going to start fresh with 4.34.2 and try again at the next version. Progress… maybe. At least I’m having fun.

@jussi_laako

have /var/lib/hqplayer/web/version.txt that

excellence. Thank you. Will get this script functional eventually.

Update. I started over and didn’t touch the /etc/hqplayer files now all works as expected including the version number and port 8088. What roller coaster. Maybe I was swapping USB sticks too fast last night But new question…
Why can’t I access the web from a different network?

[jussi_laako (Jussi Laako (Signalyst))]

@ipeverywhere

So, I’m eating your change because I blindly copied the old hqp files. I’m not sure there is a great way around this from the terminal.

It will still bind to 8088 as well, but not ideal. Normal backup restore will have similar issue. For example what Ubuntu package manager does, is that when it sees the default configuration file has changed from the last time, it asks whether you want to go with the new default or use the currently existing one.

Are you copying over the hqplayerd.service file as well, if it still manages to bind to port 80?

@ipeverywhere

Why can’t I access the web from a different network?

I’m not sure I understand what you exactly mean?

[ipeverywhere]

@jussi_laako

not sure I understand what you exactly mean

wow I’m struggling this week. I firewall myself from 8088 and completely spaced that so this a local network problem which I’ll fix. Sorry for the confusion it’s been a tough week for me.

@jussi_laako

Normal backup restore will have similar issue

It would be really excellent if you could build an “import” script. My script could call it something like:
/run/media/root-sdb2/bin/migrateconfig.py /etc/hqplayer/hqplayerd.xml /run/media/root-sdb2/etc/hqplayerd/hqplayerd.xml

Or even just migrate what is needed instead of file by file.
/run/media/root-sdb2/bin/migrateconfig.py
one command and it just moves what it needs or warns the user things are incompatible and won’t be moved. (like your apt package example)

From my initial testing of brute force which looks like this:

hqpfiles=("hqplayerd-auth.xml" "hqplayerd.xml" "hqplayerd4-key.xml")
cp /etc/hqplayer/$val /run/media/root-sdb2/etc/hqplayer/.

where $val are the files in hqpfiles

I will keep the brute force unless I see a note somewhere that says this would be destructive

Thanks!