Hydrilla on-disk data format

This page explains the upcoming format for Hydrilla site content stored in the filesystem. It refers to the upcoming Hydrilla 0.2 release.

How Hydrilla loads content

Hydrilla expects a content directory to be specified in its configuration file (under the key "content-dir"). It then processes all its direct subdirectories. If given subdirectory contains an index.json file, Hydrilla loads it (smartly ignoring "//" comments in it) and collects the definitions of site resources, pattern->payload mappings and licenses in it.

Format of an index.json

To understand the format, look into this example file with explanatory comments in it:

// SPDX-License-Identifier: CC0-1.0

// Copyright (C) 2021 Wojtek Kosior
// Available under the terms of Creative Commons Zero v1.0 Universal.

// This is an example index.json file describing Hydrilla site content. As you
// can see, for storing site content information Hydrilla utilizes JSON with an
// additional extension in the form of '//' comments support. Hydrilla shall
// look into each direct subdirectory of the content directory passed to it
// (via a cofig file option). If such subsirectory contains an index.json file,
// Hydrilla shall process it.

// An index.json file conveys definitions of site resources, pattern->payload
// mappings and licenses thereof. The definitions may reference files under
// index.json's containing directory, using relative paths. This is how scripts,
// license texts, etc. are included. Unix paths (using '/' as separator) are
// assumed. It is not allowed for an index.json file to reference files outside
// its directory.

// Certain objects are allowed to contain a "comment" field. Although '//'
// comments can be used in index.json files, they will be stripped when the file
// is processed. If a comment should be included in the JSON definitions served
// by Hydrilla API, it should be put in a "comment" field of the proper object.

// Various kinds of objects contain version information. Version is always an
// array of integers, with major version number being the first array item. When
// applicable, a version is accompanied by a revision field which contains a
// positive integer. If versions specified by arrays of different length need to
// be compared, the shorter array gets padded with zeroes on the right. This
// means that for example version 1.3 could be given as both [1, 3] and
// [1, 3, 0, 0] (aka and either would mean the same.

    // Once our json schema changes, this version will change. Our software will
    // be able to handle both current and older formats thanks to this
    // information present in every index.json file. Different schema versions
    // are always incompatible (e.g. a Hydrilla instance that understands schema
    // version will not understand version Schemas that are
    // backwards-compatible will be denoted by a different revision.
    // We will try to make schema version match the version of Hydrilla software
    // that introduced it.
    "schema_version": [0, 2],
    "schema_revision": 1,

    // Copyright of this json file. It's a list of copyright holder information
    // objects. Alternatively, "auto" can be used to make Hydrilla attempt to
    // extract copyright info from the comment at the beginning of the file.
    "copyright":  [
    // There can be multiple entries, one for each co-holder of the
    // copyright.
        // There can also be multiple years, like ["2021","2023-2024"].
        "years": ["2021"],
        // Name of the copyright holder. Depending on the situation it can
        // be just the first name, name+surname, a company name, a
        // pseudonym, etc.
        "holder": "Wojtek Kosior"

    // License of this json file. Identifier has to be known to Hydrilla. Can
    // be defined either in the same or another index.json file as a "license"
    // item. It is possible to specify license combinations, like:
    // [["Expat", "and", "Apache-2.0"], "or", "GPL-3.0-only"]
    // Alternatively, "auto" can be used to make Hydrilla attempt to extract
    // copyright info from this file's SPDX license identifier.
    "licenses": "CC0-1.0",

    // Where this software/work initially comes from. In some cases (i.e. when
    // the developer of content is also the one who packages it for Hydrilla)
    // this might be the same as "package_url".
    "upstream_url": "",

    // Where sources for the packaging of this content can be found.
    "package_url": "",

    // Additional "comment" field can be used if needed.
    // "comment": ""

    // List of actual site resources, pattern->payload mappings and licenses.
    // Each of them is represented by an object. Meta-sites and replacement site
    // interfaces will also belong here once they get implemented.
    "definitions": [
        // Value of "type" can currently be one of: "resource", "license"
        // and "mapping". The one we have here, "resource", defines a list
        // of injectable scripts that can be used as a payload or as a
        // dependency of another "resource". In the future CSS style sheets
        // and WASM modules will also be composite parts of a "resource" as
        // scripts are now.
        "type": "resource",

        // Used when referring to this resource in "dependencies" list of
        // another resource or in "payload" field of a mapping. Should
        // be consize and can only use a restricted set of characters. It
        // has to match: [-0-9a-zA-Z]
        "identifier": "helloapple",

        // "long_name" should be used to specify a user-friendly alternative
        // to an identifier. It should generally not collide with a long
        // name of some resource with a different uuid and also shouldn't
        // change in-between versions of the same resource, although
        // exceptions to both rules might be considered. Long name is
        // allowed to contain arbitrary unicode characters (within reason!).
        "long_name": "Hello Apple",

        // Different versions (e.g. 1.0 and 1.3) of the same resource can be
        // defined in separate index.json files. This makes it easy to
        // accidently cause an identifier clash. To help detect it, we
        // require that each resource has a uuid associated with it. Attempt
        // to define multiple resources with the same identifier and
        // different uuids will result in an error being reported. Defining
        // multiple resources with different identifiers and the same uuid
        // is disallowed for now (it may be later permitted if we consider
        // it good for some use-case).
        "uuid": "a6754dcb-58d8-4b7a-a245-24fd7ad4cd68",

        // Version should match the upstream version of the resource (e.g. a
        // version of javascript library). Revision number starts as 1 for
        // each new resource version and gets incremented by 1 each time a
        // modification to the packaging of this version is done. Hydrilla
        // will allow multiple definitions of the same resource to load, as
        // long as their versions differ. Thanks to the "version" and
        // "revision" fields, clients will know they have to update certain
        // resource after it has been updated. If multiple definitions of
        // the same version of given resource are provided, an error is
        // generated (even if those definitions differ by revision number).
        "version": [2021, 11, 10],
        "revision": 1,

        // A short, meaningful description of what the resource is and/or
        // what it does.
        "description": "greets an apple",

        // If needed, a "comment" field can be added to provide some
        // additional information.
        // "comment": "this resource something something",

        // One should specify the copyright and licensing terms of the
        // entire package. The format is the same as when specifying these
        // for the index.json file, except "auto" cannot be used.
        "copyright": [{"years": ["2021"], "holder": "Wojtek Kosior"}],
        "licenses": "CC0-1.0",

        // Resource's "dependencies" array shall contain names of other
        // resources that (in case of scripts at least) should get evaluated
        // on a page before this resource's own scripts.
        "dependencies": ["hello-message"],

        // Array of javascript files that belong to this resource.
        "scripts": [
            // Script name. It should also be a valid file path relative
            // to index.json's containing directory.
            "file": "hello.js",
            // Copyright and license info of a script file can be
            // specified using the same format as in the case of the
            // index.json file itself. If "copyright" or "license" is
            // not provided, Hydrilla assumes it to be the same as the
            // value specified for the resource itself.
            "copyright": "auto",
            "licenses":  "auto"
        }, {
            "file":   "bye.js"
    }, {
        "type":       "resource",
        "identifier": "hello-message",
        "long_name":  "Hello Message",
        "uuid":       "1ec36229-298c-4b35-8105-c4f2e1b9811e",
        "version":     [2021, 11, 10],
        "revision":    2,
        "description": "define messages for saying hello and bye",
        "copyright":   [{"years": ["2021"], "holder": "Wojtek Kosior"}],
        "licenses":    "CC0-1.0",
        // If "dependencies" is empty, it can also be omitted.
        // "dependencies": [],
        "scripts": [{"file": "message.js"}]
    }, {
        "type": "mapping",

        // Has similar function to resource's identifier. Should be consize
        // and can only use a restricted set of characters. It has to match:
        // [-0-9a-zA-Z]
        // It can be the same as some resource identifier (those are
        // different entities and are treated separately).
        "identifier": "helloapple",

        // "long name" and "uuid" have the same meaning as in the case of
        // resources. Uuids of a resource and a mapping can technically be
        // the same, but it is recommended to avoid even this kind of
        // repetition.
        "long_name": "Hello Apple",
        "uuid": "54d23bba-472e-42f5-9194-eaa24c0e3ee7",

        // "version" differs from its counterpart in resource in that it has
        // no accompanying revision number.
        "version": [2021, 11, 10],

        // A short, meaningful description of what the mapping does.
        "description": "causes apple to get greeted on Hydrillabugs issue tracker",

        // A comment, if necessary.
        // "comment": "blah blah because bleh"

        // The "payloads" array specifies, which payloads are to be
        // applied to which URLs.
        "payloads": [
            // Should be a valid Haketilo URL pattern.
            "pattern": "***",
            // Should be the name of an existing resource. The resource
            // may, but doesn't have to, be defined in the same
            // index.json file.
            "payload": "helloapple"
        // More associations may follow.
            "pattern": "***",
            "payload": "helloapple"
    }, {
        "type": "license",

        // Will be used to refer to this license in other places. Should
        // match the SPDX identifier if possible (despite that, please use
        // "Expat" instead of "MIT" where possible). Unlike other definition
        // types, "license" does not allow uuids to be used to avoid license
        // id clashes. Any attempt to define multiple licenses with the same
        // id will result in an error being reported.
        "identifier": "CC0-1.0",

        // This long name must also be unique among all license definitions.
        "long_name": "Creative Commons Zero v1.0 Universal",

        // We don't use "version" in license definitions. We do, however,
        // use "revision" to indicate changes to the packaging of a license.
        // Revision should be increased by 1 at each such change.
        "revision": 2,

        "legal_text": [
        // Legal text can be available in multiple forms. Usually just
        // plain .txt file is enough, though.
            // "format" should match an agreed-upon MIME type if
            // possible.
            "format": "text/plain",
            // Value of "file" should be a path relative to the
            // directory of index.json file.
            "file":   "cc0.txt"
        // If a markdown version of CC0 was provided, we could add this:
        // {
        //     "format": "text/markdown",
        //     "file": ""
        // }

        // If needed, a "comment" field can be added to clarify something.
        // For example, when definind "Expat" license we could add:
        // "comment": "Expat license is the most common form of the license often called \"MIT\". Many other forms of \"MIT\" license exist. Here the name \"Expat\" is used to avoid ambiguity."

        // If applicable, a "notice" can be included. It shall then be an
        // object with "file" field containing a path (relative to
        // index.json's directory) to a plain text file with that notice.
        // "notice": {
        //     "file": "license-notice.txt"
        // }
        // This is needed for example in case of GNU licenses (both with and
        // without exceptions). For example,
        // "GPL-3.0-or-later-with-html-exception" could have the following
        // in its notice file:
        // This program is free software: you can redistribute it and/or
        // modify it under the terms of the GNU General Public License as
        // published by the Free Software Foundation, either version 3 of
        // the License, or (at your option) any later version.
        // This program is distributed in the hope that it will be useful,
        // but WITHOUT ANY WARRANTY; without even the implied warranty of
        // GNU General Public License for more details.
        // As a special exception to the GPL, any HTML file which merely
        // makes function calls to this code, and for that purpose
        // includes it by reference shall be deemed a separate work for
        // copyright law purposes.  If you modify this code, you may extend
        // this exception to your version of the code, but you are not
        // obligated to do so.  If you do not wish to do so, delete this
        // exception statement from your version.
        // You should have received a copy of the GNU General Public License
        // along with this program.  If not, see
        // <>.

Updated by koszko 10 days ago · 17 revisions