Source Development/Tutorial

This document will provide a step-by-step guide for creating a (trivial) new Source in MS alongside describing what aspects of the Source need to be implemented based on the service you use. Before using this document you should review Common Development.

Scenario

You are the developer of a fancy, new self-hosted web-based media player called Cool Player. Cool Player has a slick interface and many bells and whistles, but most importantly it has an API. The API:

Has an unauthenticated health endpoint at /api/health that returns 200 if the service is running properly
Has authenticated endpoints that require a user-generated token in the header Authorization MY_TOKEN
- Has a /api/recent endpoint that lists recently played tracks with a timestamp
- Has a /api/now-playing endpoint that returns information about the state of the player like current track, player position in the track, etc...
Cool Player is by default accessed on port 6969
Your personal instance of Cool Player is hosted at http://192.168.0.100:6969 and the api is accessed at http://192.168.0.100:6969/api

Because there is an API that MS can actively read this will be a polling Source where MS sends requests to Cool Player to get scrobble information -- as opposed to an ingress Source like Webscrobbler that uses webhooks from the service to send data to MS.

Minimal Implementation

Define and Implement Config

We will create a new config interface for Cool Player using the Common Config and tell MS it is a valid config that can be used.

MS uses Zod to define schema objects and the infer the config type from this schema.

The eaiest way to use this for our new config is focus on implementing only the data and options schemas, then assigning those to a generic source config schema.

Create a new file for your config and implement a data schema, this is the required config for the Source to operate:

Loading ...

Basic options and the complete config shape:

Loading ...

If your Source should also be configurable via ENV then implement a separate shape for ENV (reusing shape from data/options) as well as an envSchemas object that is used by MS for validation, generating ENV config docs, and transforming common ENV keys for your Source:

Loading ...

Finally, add a shape for use with AIO config:

Loading ...

Add to Config Map

Next, we add our newely created config Types and zod schemas to several data structures that MS uses for typing Sources, generating documentation, and parsing/validating config data.

In sources.ts, import coolPlaySourceConfigSchema (for File) and coolPlayerSourceAIOConfigSchema (for AIO) and add them to the ends of their respective union types:

sources.ts
export const sourceConfigSchema = z.union([
    // ...
    appleMusicSourceConfigSchema,
    coolPlaySourceConfigSchema
]);

export type SourceConfig = z.infer<typeof sourceConfigSchema>;

export const sourceAIOConfigSchema = z.union([
    // ...
    appleMusicSourceAIOConfigSchema,
    coolPlayerSourceAIOConfigSchema
]);

In sourcesMap.ts add coolplayer to SourceTypeConfigMap for type narrowing and sourceConfigSchemaMapAsync to enable dynamic importing for schema validation.

sourcesMap.ts
// ...
export interface SourceTypeConfigMap extends Record<SourceType, [CommonSourceConfig, SourceAIOConfig, Partial<Pick<CommonSourceConfig, 'data' | 'options'>>]> {
  // ...
  coolplayer: [CoolPlayerSourceConfig, CoolPlayerSourceAIOConfig, Partial<Pick<CoolPlayerSourceConfig, 'data' | 'options'>>];
}

// ...

export const sourceConfigSchemaMapAsync: { [K in keyof SourceTypeConfigMap]: () => Promise<[ZodType<SourceTypeConfigMap[K][0]>, ZodType<SourceTypeConfigMap[K][1]>, EnvSourceSchema<ZodObject, SourceTypeConfigMap[K][0]>]> } = {
    // ...
    coolplayer: async () => {
        const {coolPlayerSourceConfigSchema, coolPlayerSourceAIOConfigSchema, envSchemas } = (await import('./coolplayer.ts'));
        return [coolPlayerSourceConfigSchema, coolPlayerSourceAIOConfigSchema, envSchemas]
    },
}

Create CoolPlayer Source

Now we will create a new Source inheriting from AbstractComponent that:

accepts our config interface
implements a function to transform CoolPlayer's track data into a PlayObject
implements required stages
implements required methods to get current player state and/or now playing track

First we create a new Source called CoolPlayerSource and setup our constructor to accept the config as well as setting any behavior overrides.

Loading ...

Implement Stages

Next we will implement the Stages required to get CoolPlayerSource running.

Build Data

First we implement the Build Data Stage. We will check the baseUrl property includes the necessary prefix and then parse it to a normalized data structure we can use elsewhere.

Loading ...

Check Connection

Second we will implement the Check Connection Stage:

Loading ...

Test Auth

Finally, we will implement Auth Test Stage:

Loading ...

Implement Play Object Transform

Now that Source initialization is taken care we can move on to implementing how our Source handles activity data from Cool Player.

First, we create a static function that is used to take the track data returned from Cool Player's API and return a standard PlayObject.

Loading ...

Implement Polling

Next, we implement the functions needed for MS to get the actual data from Cool Player and do something with it.

The majority of Sources MS monitors primarily operate as a source of truth for a music player rather than a played music history.

To this end, MS implements a state machine that emulates the behavior of a music player in order to keep track of when a song you are listening to should be scrobbled. It does this by monitoring the "currently playing" track reported by a Source's service, with varying degrees of accuracy depending on what information is returned from the service. The state machine is implemented in MemorySource which our CoolPlayerSource inherits from.

For a polling Source to work properly we need to implement a function, getRecentlyPlayed, that returns PlayObjects or PlayObjects + Player data, that are "new". These are then checked against previously "discovered" Plays and their timestamp to determine if they should be surfaced to Clients to scrobble.

To take advantage of the MemorySource state machine we will additionally use processRecentPlays from MemorySource inside getRecentlyPlayed. We pass track and/or player state returned from the Source service to processRecentPlayers. It then takes care of deriving Source player state based on how this data changes over time. The advantage to using processRecentPlays is that our Source service does not necessarily need to pass any player information -- as long as the track info has a duration we can more-or-less determine if it has been played long enough to scrobble.

Loading ...

There are other types of Sources that can be implemented based on the granularity of data returned or the type of data returned. See further below, Other Source Types.

Initialize Source from Config

The last step for this implementation is to add your new, concrete CoolPlayerSource class to the Source builder class that handles parsing user config.

When MS starts it reads all configs and determines which Source to build based on the configs found. We need to tell it to build a CoolPlayerSource when a coolplayer config type is found.

We modify ScrobbleSources.ts to add CoolPlayerSource as a case in the addSource function:

src/backend/sources/ScrobbleSources.ts
// ...
export default class ScrobbleSources {
  // ...
  addSource = async (sourceType: SourceType, strongConfigs: CommonParsedConfig[], defaults: SourceDefaults = {}) => {
      switch (sourceType) {
          case 'spotify':
          // ...
          //
          // add a dynamic import for CoolPlayersource
          case 'coolplayer': {
              const CoolPlayerSource = (await import('./CoolPlayerSource.ts')).default;
              await this.instantiateSources('coolplayer', strongConfigs, defaults, CoolPlayerSource);
          } break;
          default:
              break;
      }
      // ...
  }
  // ...
}

Congratulations! Your new Cool Player Source is implemented and ready to be built by MS with the config your defined.

Further Implementation

Backlog

To have your Source try to scrobble "missed" tracks when MS starts up the Source's service must be able to provide:

track information
timestamp of when the track was played

In your Source implement getBackloggedPlays and set setting in constructor indicating it has backlogging capabilities:

src/backend/sources/CoolPlayerSource.ts
import request from 'superagent';
import {
  PlayObject,
} from "../common/infrastructure/Atomic.js";
// ...
export default class CoolPlayerSource extends MemorySource {

  // tell MS it should try to get backlogged tracks on startup
  override canBacklog: boolean = true;

  constructor(/* ... */) {
    super(/* ... */);
    // ...
  }

  // ...

  protected getBackloggedPlays = async (options: RecentlyPlayedOptions): Promise<PlayObject[]> => {
    try {
      const resp = await request
              .get(`${this.urlData.url.toString()}/history`)
              .set('Authorization', `Token ${this.config.token}`);

      // assuming list from body looks like track info returned in 
      // "Implement Play Object Transform" section
      const {
        body = []
      } = resp;

      return body.map(x => CoolPlayerSource.formatPlayObj(x));
    } catch (e) {
      throw new Error('Error occurred while getting recently played', {cause: e});
    }

  }
}

Other Source Types

There are some scenarios where polling and/or state machine is not the right tool to handle determining if incoming data should be scrobbled:

The Source service handles scrobble threshold internally, the data being received should always be scrobbled (WebScrobbler, Plex, Tautulli, Listenbrainz, Last.fm)
You prefer to handle the scrobble determination yourself

Music History Source

If the Source is still polling but the track returned should always be scrobbled if not already seen IE the Source service is a music history source (Listenbrainz, Last.fm), rather than a music player, then simply indicate to MS the source of truth type by setting it in the constructor. The state machine will always return a track if it is new and not seen, regardless of how recently it was seen:

src/backend/sources/CoolPlayerSource.ts
import { SOURCE_SOT } from "../../core/Atomic.js";
// ...
export default class CoolPlayerSource extends MemorySource {

  // tell MS it should immediately scrobble any new, unseen tracks from the upstream service
  override playerSourceofTruth = SOURCE_SOT.HISTORY;

  constructor(/* ... */) {
    super(/* ... */);
    // ...
  }
}

Remember to also set meta.parsedFrom for your Play objects.

Non-Polling Source

Ingress Sources (like Plex, Tautulli, Webscrobbler, Jellyfin) do not having a polling mechanism because the upstream service contacts MS when there is an event, rather than MS contacting the upstream service.

For these Sources you will need to implement endpoints in src/service/api.ts and corresponding files. See the existing Sources in the project as references for how to do this.

You may still wish to use the state machine MemorySource (like Jellyfin) if the events received are not "scrobble" events but instead of implementing getRecentlyPlayed you will implement your own function in your Source class, like handle(), that receives data and then uses processRecentPlays.

After new plays have been determined see the next section for how to scrobble...

Basic Source

At the core of a Source that implements AbstractSource's functionality is the ability to Discover and Scrobble plays.

These functions are not seen in the MVP CoolPlayerSource because they are automatically done by the polling functionality after being returned from getRecentlyPlayed.

Discovery

A Source keeps track of all the "plays" that are determined to be valid for scrobbling. When a play is valid it is checked to see if it has already been "discovered" by comparing the track info and timestamp of the play against already discovered plays. This prevents duplicate scrobbling by using the Source's own data and simplifies scrobbling for Sources by allowing your implementation to "always" ingest track data without having to worry about whether its new or not -- AbstractSource and discover() will take care of that for you.

src/backend/sources/MyBasicSource.ts
export default class MyBasicSource extends AbstractSource {
  handle(somePlay: PlayObject) {
    // if the track is "new" and not seen before it is returned in the discovered list
    // we then know it is OK to be sent to Clients for scrobbling
    const discovered: PlayObject[] = this.discover([somePlay]);
  }
}

This additionally will be surfaced to the user in the Dashboard in the "Tracks Discovered" page.

Scrobbling

After a play is verified to be discovered we can then scrobble it. This will emit the plays to the ScrobbleClients service which then disseminates the play to all Clients that were configured to listen in the Source's config.

src/backend/sources/MyBasicSource.ts
export default class MyBasicSourceSource extends AbstractSource {
  handle(somePlay: PlayObject) {
    const discovered: PlayObject[] = this.discover([somePlay]);
    // emit plays that can be scrobbled by clients
    this.scrobble(discovered);
  }
}

If your service only emits an event when a play is scrobbled you can technically skip using discover() but it is good practice to use it unless you have a very good reason not to.

note

Using scrobble() does not guarantee a track is actually scrobbled! The Scrobble Clients also check the play against their own "recently scrobbled" list to prevent duplicates.

Scenario​

Minimal Implementation​

Define and Implement Config​

Add to Config Map​

Create CoolPlayer Source​

Implement Stages​

Build Data​

Check Connection​

Test Auth​

Implement Play Object Transform​

Implement Polling​

Initialize Source from Config​

Further Implementation​

Backlog​

Other Source Types​

Music History Source​

Non-Polling Source​

Basic Source​

Discovery​

Scrobbling​

Scenario

Minimal Implementation

Define and Implement Config

Add to Config Map

Create CoolPlayer Source

Implement Stages

Build Data

Check Connection

Test Auth

Implement Play Object Transform

Implement Polling

Initialize Source from Config

Further Implementation

Backlog

Other Source Types

Music History Source

Non-Polling Source

Basic Source

Discovery

Scrobbling