Let's develop a platform for extension distribution [Repository open]

Tinkering Roon API
1

In the RoPieee thread the suggestion was made by @spockfish to come up with a convention for extension distribution, to allow the creation of a generic extension installer.

What follows is the current status, this is of course open for discussion.

The repository entry

The backbone behind the distribution is npm. An extension gets an entry in a global extension repository in json format. An entry contains the fields that are shown in the below example:

{
  "author": "The Appgineer",
  "display_name": "Alarm Clock",
  "description": "Roon Extension to start or stop playback on a specific zone at a specific time",
  "repository": {
    "type": "git",
    "url": "https://github.com/TheAppgineer/roon-extension-alarm-clock.git"
  }
}

The first three fields are shown to the user by an extension installer, the last one is used to install the extension. Updates can be automatic and do not require an update of the above information.

The repository

The repository is hosted on GitHub:

If you are an extension developer and you want your extensions included, please give me a pull request!

The platform layering

roon-extension-manager
This is the front-end towards the lower layers. There is no state information of the managed extensions in this layer and it can therefor easily be replaced by another front-end. In the current implementation the functionality behind the auto-update time hasn’t been implemented yet, I’m working on it right now.

node-api-extension-installer
This layer wraps npm commands in a Node API. I could not find an existing API for this that handles packages outside the npm repository (Roon extensions currently have to be accessed via a git url). Installed packages and available updates are all queried via standard npm commands.

node-api-extension-runner
This is a child node implementation for starting and stopping extensions. It is rather limited in its functionality, it e.g. doesn’t store the state of extensions, preventing an auto restart of running extensions after a reboot. The plan is to come up with OS optimized implementations of this layer.

2 Likes
2

I’m wondering if we’re gonna use specific releases then it won’t be possible (easy) to upgrade the extension…

Wouldn’t it be an idea to suggest a branche on the specific github page with a fixed name (‘roonextension_stable’) so that the host can pull for updates?

3

ah ok. on second thought, if you can supply a git url in the ‘roonextension.json’ file then that would serve that purpose i guess

4

Maybe it is indeed easier to clone a repository instead of using the archive. The version information can then be left out:

{
  "author": "The Appgineer",
  "friendly_name": "Alarm Clock",
  "description": "Roon Extension to start or stop playback on a specific zone at a specific time",
  "git": "https://github.com/TheAppgineer/roon-extension-alarm-clock.git"
}

When the repository is cloned the current version can be read from the ‘package.json’ file that should be in the root of the repository. Updates can be automatic and do not require an update of the above information.

2 Likes
5

I wonder if we can come up with, what I would like to call, a Meta Extension.

This extension should be pre-installed on RoPieee, other end points and servers and it allows the user to install other extensions. The advantage of this approach would be that installation and configuration of extensions can all be done from within the Roon UI.

I’m writing some experimental code at the moment to see if this idea is feasible.
To be continued…

2 Likes
6

ok cool. let me know if you’ve got a rough plan then I can have a look at it.

7

@Jan_Koudijs tomorrow I’ve planned some time to see what we can do. I’ll just start with having a look at your extension, see how I can integrate it and how we can make that more generic.

I’ll keep you posted.

8

Thanks for taking time for this. If you can integrate the Alarm Clock extension and let it auto start at power-up it would be a nice first step.

I’m still working on the meta-extension idea and spent time on finding a way to capture extension installation in the settings UI. No show-stoppers till now. The next more critical step is to see if I can clone a repository and start a child node process from it.

Here is a screenshot to give you an impression of the current status:

extension_installer.jpg551×541 36.7 KB

9

Ok @Jan_Koudijs I’ve got your extension running and auto starting as well. Installing is done from a script, including the auto start process.

For now I think that the easiest way would be that I integrate your timer as package in RoPieee; that would then be based on ‘whitelisting’ the extension so that it conforms to certain definitions and does not do any ‘harm’ to RoPieee’s primary task: running RoonBridge.

Lot’s of stuff to work out though: logging is an issue (the Roon Node.js components are rather verbose) and there should be an update scenario. That could very well be a simple ‘git pull’, but something to think about.

1 Like
10

Nice that you got it running that fast.

When not debugging I just redirect stdout and stderr to /dev/null. I don’t think it is feasible or necessary to keep logs on an endpoint like RoPieee.

I’m making good progress in the development of the extension that can be used to install other extensions from a global repository. I will create a separate thread for this Roon Extension Installer.

So what I like to propose is that you integrate this installer extension and let it start at power-up. From that point on the installer can also take care of updates (using git pull). By containing most of the functionality in the extension itself I hope to come up with something that can be easily integrated in endpoints and servers.

I hope you like this idea, just let me know.

11

I see where you going. You want a generic (as least as much as possible) solution of running and managing extensions.

I’m curious what you’re planning for managing the individual extensions. So you’re going to take care of start/stopping etc. as well? I’m not sure I’m a big fan of that approach.

Right now I’ve created a template systemd unit file for the extensions; this gives fine-grained control of the extensions and makes sure that an extension can not ‘ruin’ the system. From a RoPieee perspective that’s a rather strong requirement.

Furthermore your extension should provide enough ‘freedom’ for integrators (like myself) to tailor the solution to certain environments… difficult but fun :wink:

Let me know If I can help in any way!

Regards,

12

That’s the goal.

What can be done by the host system and what by the extension that’s what we have to find out. My current strategy is to start off by placing most of the functionality in the extension and then discover what doesn’t work, or is a problem for integrators. Those parts can then by placed outside the extension.

The installer extension can keep a list of running extensions and start them when the installer itself starts. The extensions then run as a child process of the installer. If you have reasons why I shouldn’t do this, please let me know. This is where I have my doubts as well, at least it won’t be very scalable.

There is control over the installable extensions by having criteria for inclusion in the repository. A criteria can be a thread on this forum so users can get support.

This is indeed very important. If a certain extension turns out to be ‘highly demanding’ for a host then it could be marked as a ‘server only’ install.

I have some questions about your current integration. Do you clone the git repository and install the dependencies as part of the extension installation, triggered from the web interface?

Do you have a watchdog mechanism, taking care of extension restart in case it unexpectedly terminates?

13

Right now my script clones the git repository, runs an ‘npm install’ inside the directory and starts a systemd unit. This systemd unit is a template for all extensions and because it’s controlled by systemd you can do all kinds of things like restarting on failure, control how often it will restart on failure, manage resources etc.

For Linux systems I would seriously recommend to make this the solution for running the extensions as systemd is default on all major distro’s.

With respect to the solution: is it an idea to create a node js API that takes care of this? It’s then possible to create a platform abstraction (Linux and Windows to start with), and your meta extension would then talk against this API. Installing a new extension ('basically a git clone and some service stuff) will be taken care of by the API. This component could also hold a whitelist of available extensions that it pulls from github itself on regular moments. This list with extensions can be queried via the API, making it visible in your extension. The user can then select ‘install’ (or ‘uninstall’ for that matter) and the component takes care of the rest.

My 2 cts.

14

I like your layering proposal.

So if I understand you correctly we have the following three layers (ordered from lower to upper):

OS Specific Layer
This layer takes care of starting and stopping the installed extensions, optimized for the host OS. For Linux this is implemented via systemd units, for Windows and Mac OS X separate implementations have to be created.

Middleware Layer
This layer is implemented as a node.js API. It accesses a global extension repository and takes care of installing and uninstalling extensions. It accesses the OS Specific Layer to start and stop the extension execution.

Front End Layer
This layer provides a frontend for the Middleware layer, giving the user the possibility to manage extensions. This can be implemented as a Roon Extension or can be included in a web interface like the one provided by RoPieee.

If this is the plan then I can split my code into an API and a frontend part and we can define the interface between the API and the OS layer.

Do you have a suggestion for the interface between the node.js code and your systemd template? I’m not too experienced in this area.

1 Like
15

I was thinking that the OS specific layer is a node.js API in itself:

list_extensions
stop_extension
start_extension
status_extension

Something along that line. These calls can then be accessed by the middleware layer.
for Linux I can get this working. we than can build it up and get it to work and flesh out all the details. Along the line I’m pretty sure someone jumps on board to get it running on Windows as well.

16

I guess I don’t understand the issue you are trying to solve. All of your data points are duplicating data already expressed in the package.json. Any extra bits of config data a Roon Extention should be expressed in the package.json in an object labeled roon which would be following NodeJS best practices.

{
  "name": "my-roon-extention",
  "version": "0.0.0",
  "description": "Something amazing...",
  "license": "MIT",
  "repository": "haysclark/pandora-client",
  "roon": {
    "key": "value"
  }
}

The package.json already has it’s own official repository syntax.

"repository": {
    "type": "git",
    "url": "git://github.com/jieter/marinetraffic"
  },

Even something like requiring a specific RoonCore version likely belongs in the package.json’s engines object.

{ 
  "engines": {
    "node": ">=6",
    "rooncore": ">=1.0.13"
  }
}
17

I wrapped the functions that have a match in my current implementation in an API.
I will write my code against this API if this will also work for you.

The easiest way to share it was by putting it on GitHub.

Just give me a pull request if you want something changed.

2 Likes
18

The reason behind this is that I don’t want to clone a Git repository only to get access to a package.json file. But maybe your point is that the extension repository can just be a collection of package.json files.

What I will do, at least, is have a good look at the npm documentation and make sure that I create something that conforms to the format. Thanks for pointing this out.

19

You can access a single file in GitHub via a single HTTPS curl call, thus one could easily CURL the url and then parse the file.

You want to make a registry for all of the Roon Extensions?

20

Out of curiosity, how familiar are you with NodeJS and it’s ecosystem of tools as well as authoring Web applications? Others have mentioned being new to NodeJS, so I am trying to ask in earnest. (I’m not trying to be a jerk.)

Have you looked at NPM’s website and look how other companies just use naming conventions? https://www.npmjs.com/

The other technology which has become critical to distributing modern server environments is Docker. Are you familiar with Docker and/or DockerCompose? https://www.docker.com/

In general, I wouldn’t expect the Roon Community to tackle a problem on the scale of distributing a NodeJS/Javascript application. :slight_smile: Docker is the only solution to date which enables the safe installation of Databases and other low-level utilities without the huge resource hit of VM’s.

It sounds like Roon is hoping to store all of the data for a given extensions in its own API supplied data store. So, NPM should be powerful enough to do all the heavy lifting to install the needed programs for a given Roon Extension. NPM can install other needed like Bower, etc. If NPM is the end goal for RoonLabs, then NPM itself would be the correct distribution system and repository for Roon itself. All Roon would need to specify is that all extension use the same ‘nvm’ command if Roon were to install and them and launch them. Of course, this is easier said than done… (see nombom).

RoonLabs might not have published their Extensions officially yet on NPM because you can no longer delete Project once published. However, they still might assume that NPM will be the official repository for Roon Extensions. But that is just a guess on my part.

-Hays