A proposal for standarising a subset of JSON-LD

— Stéphan Kochen, 2018-10-20

One of my interests of late has been ActivityPub, a standard for building social networks as a federated system, popularized by Mastodon. If you haven't seen it yet, I have an experiment running called castling.club, which is an implementation of an ActivityPub server that can set up a game of chess between you and another user in the ‘Fediverse’.

ActivityPub relies heavily on JSON-LD, a standard that brings the ‘linked data’ / ‘semantic web’ world to JSON. The use of JSON-LD has been something of a discussion point, which I'd like to offer my view on here.

ActivityPub lends itself well to being part of the semantic web. As a protocol, it describes content, plus connections and interactions between content and people.

In very practical terms, though, ActivityPub is currently being used as a network protocol, with many different implementations in different languages and frameworks. These implementations often just want to exchange content and actions, and treat everything as plain JSON as much as possible.

(The rest of this post will go into more technical detail about these standards.)

“JSON-LD is just plain JSON”

It's one of the selling points of JSON-LD, that it's still plain JSON. This is true when you 'upgrade' existing producers and consumers, who already know about eachother, because all that is required is to add some context metadata to map properties and values into the model of the semantic web. Your apps can ignore this metadata and keep going as is.

I believe this promise falls short in the case of ActivityPub, though, or any federated system that tries to build directly on JSON-LD. Here, producers and consumers are different implementations that have taken the freedom of JSON-LD to format documents any way they want, through namespacing and aliasing. Now, every consumer needs to be a full implementation of JSON-LD in order to understand anything from its peers.

JSON-LD is a complex specification

For several weeks now, I've been building an implementation of the JSON-LD 1.0 spec as a spare time project, focusing on the expansion algorithm. This implementation is now passing almost all expansion tests.

The time investment alone tells that this is no simple spec to implement. I believe this eventually creates a heavy reliance on only a few implementations, which doesn't sound like a healthy goal.

JSON-LD 1.0 had some arbitrary limitations (e.g. no lists of lists) that are being addressed in JSON-LD 1.1, but unfortunately at the cost of even more complexity.

ActivityPub uses a very limited subset

The important parts of JSON-LD that ActivityPub implementations actually deal with are related to namespacing, and through it versioning and extensions. This is a subset roughly equal to XML namespaces in scope.

The remaining functionality offered by JSON-LD deals with mapping JSON to the semantic web. Keywords like @type, @list, @reverse, even @id are all details often of no interest to ActivityPub consumers, but constitute the bulk of the complexity in the JSON-LD spec.

Essentially, JSON-LD adds new type information to JSON documents. This puts JSON-LD as a whole closer to a schema description than just a namespacing extension to JSON.

Combine this with the absolute most killing feature of the spec in terms of operational complexity: remote references. Your app trying to interpret a JSON-LD document from an unknown producer needs all context metadata, including any remote references, or it cannot procede.

All the other complexity can be lumped under algorithmic complexity, but remote context loading adds a slew of potential network and security issues. This is no good in ActivityPub, which is about lots of people exchanging lots of content at potentially very high rates.

Let's use just a subset

I believe it makes sense to define a basic subset of JSON-LD that simplifies the implementation in network protocols. Defining a subset means current implementations can ‘drop down’ to conform with it, while keeping all the JSON-LD detail in their documents intact. With enough traction, we would ease the implementation for ActivityPub as a whole.

Here's what I propose, let's call it JSON-NS, because we're approaching something similar to XML-NS:

  • No errors. Anything that appears invalid is simply ignored. This is especially true for @context, which can then mix in definitions for a full JSON-LD processor.

    A misinterpretation is expected to happen in the application itself. The output document will simply be malformed to the application.

  • Minimal processing of values and names. The output document structure should mostly resemble that of the input, because we shouldn't be thinking about an RDF data model in the background.

    As much as possible, let's also treat strings as strings, and not depend on other complex standards. Even something as ‘simple’ as a URL / URI / IRI has many different meanings on the internet, and often implementations rely on different libraries that don't agree with eachother.

  • No value merging. For example, the standard ActivityPub context defines content and contentMap names which both map to the same IRI, and are merged in JSON-LD. The underlying RDF data model als means that any value can suddenly become an array in the output.

    In JSON-NS, structure is preserved, no such merging happens, and only one of content or contentMap can be used. Preferably, we settle on using just one of them everywhere.

  • No recursion in definitions, context term interdependence, or having to deal with cyclical references within contexts.

  • And, of course, no external contexts. In order to get there, we also want an inline context that is as simple as possible, so that humans may parse it as well, without having to read a spec.

The algorithm

  • We use the following terminology very specifically:

    • ‘Keyword’: a string that starts with @.
    • ‘Absolute IRI’: a string that contains : and doesn't start with @.
    • ‘CURIE prefix’: a non-empty string that doesn't contain : nor starts with @.

  • A context is a structure containing:

    • An optional default namespace, which is an absolute IRI.
    • A default language, a string which defaults to the empty string.
    • A mapping of CURIE prefixes to absolute IRIs.
    • A mapping of property names to aliased names.
    • A mapping of property names to container mappings.

  • Name expansion is as follows:

    1. If the name starts with @, it is dropped.

    2. If the name matches a CURIE prefix in the context (prefix:suffix), the result is the defined namespace and CURIE suffix concatenated.

    3. If the name is an absolute IRI already, the result is the name unaltered.

    4. If a default namespace is defined, the result is that namespace and the name concatenated.

    5. Otherwise the name is dropped. (Aliases are not part of normal expansion.)

    Optionally, a processor may do postprocessing before returning, if handling full IRIs is inconvenient in the programming environment. The reference implementation does this by introducing a 'target context', which is simply a bunch of rules to again compact IRIs to CURIEs.

  • Any object can have a @context property, which introduces new context for the object itself and any nested values, on top of context that was inherited.

    This property is processed before other properties in an object, and dropped from the output.

    The value may be an array, and if not, the processor should pretend the value is wrapped in an array.

    The elements are processed in order. An element may be null, which clears any earlier context, including inherited context. Otherwise it is an object; any other type is ignored. Objects are processed as follows:

    1. The @vocab property sets the default namespace. The value is an absolute IRI or null, otherwise ignored. The default is null, which means unrecognized names are dropped in expansion.

    2. The @language property sets the default language for internationalised properties whose value doesn't specify one. The value is a string or null, otherwise ignored. null is treated as the empty string, which is also the default. (Non-empty values should be BCP47 language codes, for JSON-LD compatibility.)

    3. Other keyword properties are ignored.

    4. If the property value is a string, it defines a namespace. The property name is a CURIE prefix and the value an absolute IRI, otherwise the property is ignored.

    5. If the property value is an object, it annotates a document property matching that exact name. The object can contain:

      • A property @id, which defines an alias. Its value is a string that can't be a keyword, or the alias is ignored.

      • A property @container, which defines a container mapping. Its value is a string, or the container mapping is ignored.

    6. If the property value is null, it clears any definitions associated with the property name. (These are: namespace definitions, aliases, and container mappings)

  • The algorithm recursively processes any type of value, acting like a 'deep clone', while making the following alterations to the document:

    1. An @id property is copied to the output if it contains an absolute IRI, otherwise it is dropped.

    2. An @type property is an array, or the processor pretends the value is wrapped in an array. The output is always a non-empty array. Each element is a name that is expanded using the current context, or dropped if it is invalid.

    3. Other keyword properties are dropped.

    4. The output name of a property is determined as follows:

      1. First look up if an alias exists for the property name, and continue with its value if so.

      2. Expand the name using the current context, or drop the property if it is invalid.

    5. If the original property name has a container mapping @language, the value is normalised to a language map:

      1. If the value is a string, it is normalised to: { "<default language>": "<value>" }

      2. If the value is an object, it can only contain properties with string values. Other properties within are dropped. (The property names within should be BCP47 language codes, for JSON-LD compatibility.)

      3. Other values are ignored, and the property dropped.

    6. Otherwise, the algorithm recurses on the property value to produce the output property value.

An example

We'll take this toot as an example: https://mastodon.social/@Gargron/100211283409260120

It can be encoded in JSON-NS as:

{
  "@context": {
    "@vocab": "https://www.w3.org/ns/activitystreams#",
    "ostatus": "http://ostatus.org#",
    "contentMap": { "@id": "content", "@container": "@language" }
  },
  "@id": "https://mastodon.social/users/Gargron/statuses/100211283409260120",
  "@type": "Note",
  "url": "https://mastodon.social/@Gargron/100211283409260120",
  "to": "https://www.w3.org/ns/activitystreams#Public",
  "cc": "https://mastodon.social/users/Gargron/followers",
  "published": "2018-06-15T22:47:15Z",
  "attributedTo": "https://mastodon.social/users/Gargron",
  "contentMap": {
    "": "<p>Discourse</p>",
    "gl": "<p>Discourse</p>"
  },
  "attachment": {
    "@type": "Document",
    "mediaType": "video/mp4",
    "url": "https://files.mastodon.social/media_attachments/files/004/507/972/original/653d2d21eda95b18.mp4"
  },
  "sensitive": false,
  "tag": [],
  "ostatus:atomUri": "https://mastodon.social/users/Gargron/statuses/100211283409260120",
  "ostatus:conversation": "tag:mastodon.social,2018-06-15:objectId=38093055:objectType=Conversation"
}

Some notes from this example:

  • Compared to the original, this document lost type information, such as which properties are references to other subjects (@id objects). This can be restored by extending the context with directives ignored by a JSON-NS processor, but used by full JSON-LD processors, typically a remote context reference. (This is why we want to still allow arrays in @context.)

  • Not being able to alias id to @id and type to @type is a deliberate decision. This is a minor inconvenience, but makes processing simpler, reduces confusion, and keeps the context small.

  • Internationalisation is the trickiest part.

    The above uses contentMap, and defines the alias and @language container mapping like the ActivityPub JSON-LD context. The processor guarantees the output is a language map.

    But it is also valid to use content, which in JSON-NS would be a string in the output.

    Applications would have to account for both possibilities, or we'd have to somewhere spec that ActivityPub on JSON-NS is restricted to only contentMap. I like strictness, so I prefer the latter.

    The situation is not optimal either way, because JSON-LD allows internationalisation almost anywhere, such as mediaType if you so desired.

Compatibility

A plain JSON-NS document like the above does not participate in the world of linked data, because all links were lost when type information was abandoned. Using JSON-NS without JSON-LD is okay for some niche purposes perhaps, but it doesn't look like there's a demand for JSON-NS by itself, given that we've all been building plain JSON APIs for quite some time now instead.

The sole purpose of JSON-NS would be to encode, in a network protocol, a simpler document processor than full JSON-LD. The documents should still contain all the additional context metadata of JSON-LD.

Reference implementation

I've created a reference implementation with some tests to get things going. Feel free to (ab)use the issue tracker there for discussion.