Advanced Concepts

This section goes into more detail on some the subjects covered in Core Concepts, as well as introducing additional advanced topics.

Events#

The principal objects in the DataTrails platform are Events. These are the records that represent the collective ‘Golden Thread’ of evidence contributed by all stakeholders about a particular thing.

That ’thing’ really can represent anything: a file, a piece of data, a physical thing or even a business process. As long as shared accountability needs to be traced and trustworthy, Events can be recorded about it. If there are moments that matter they can be committed to the immutable audit log.

What defines a moment that matters? It’s all about the use case: if you think you might need to prove something in a multi-party dispute later, chances are you can save a lot of time and stress by committing it to the ledger. Simply looking and knowing the current state of things isn’t enough: sure, it has software version 3.0 now but when was that released? Before the major incident? After the major incident? This morning before the support call? By recording events into an immutable audit trail, questions relating to that fact can be answered.

DataTrails ensures complete and tamper-proof lineage and provenance for all Asset attributes by enforcing a simple rule: Events cannot be modified once published! No shredding, tampering or backdating is possible without leaving an undeniable trace in the log.

Event Attributes#

DataTrails Events are essentially very simple: a collection of attributes that describe the Event expressed as a JSON document. The power of the system comes from the fact that those attributes come with complete traceable provenance and are guaranteed to appear the same to every stakeholder, creating a single source of truth for shared business processes.

DataTrails is not opinionated about Event content, meaning that attributes can trace anything deemed important to participants. Much like #hashtags on social media platforms, they can be invented by anyone at any time, but once an attribute has been seen once it will be fully traced from that point on.

Asset Container#

Note: Assets will soon be deprecated for a more flexible and powerful concept of Trails.

The change is subtle, but where Assets only allow Events to be registered against a single thing or theme, Trails will enable Events to be free of any such restriction leading to more natural expression of what happened when or who said what about what.

To prepare for this change, when writing code integrations be sure to focus on Event attributes and not mutable asset_attributes. This will ensure best performance and minimal code changes to take advantage of the newer API. Trails will still support simple properties like types, descriptions and thumbnails for search and grouping purposes.

Events are registered into collections called Assets. These Assets might represent an individual thing, a class of things, or something more abstract like ‘all Events for this day’. Some care should be taken in designing the scheme for this as it will aid in simplifying sharing policies.

A simple Asset might look like this:

   {
      // Fixed global identity for this Asset
      "identity": "assets/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",

      // The set of tracked and traced attributes
      "attributes": {
        // arc_* are understood and processed by DataTrails
        "arc_display_type": "Top Trump",
        "arc_display_name": "Opel Kadett Rallye 1900",
        "arc_description": "Opel Kadett Rallye 1900 Top Trump card",
        "arc_serial_number": "5a",
        "arc_home_location_identity": "locations/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",

        // attribute with "arc_attribute_type": "arc_attachment" is a reference to a BLOB attached to the Asset
        "arc_primary_image": {
          "arc_attribute_type": "arc_attachment",
          "arc_blob_hash_value": "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
          "arc_blob_identity": "blobs/1754b920-cf20-4d7e-9d36-9ed7d479744d",
          "arc_blob_hash_alg": "SHA256",
          "arc_file_name": "somepic.jpeg",
          "arc_display_name": "arc_primary_image",
          },
        
        // All others are user-defined attributes
        "bhp": "145",
        "top_speed": "190",
        "weight": "860",
        "engine_capacity": "1980"
      },

      // Behaviours define the set of APIs that can be used to interact with the Asset
      "behaviours": [
        "RecordEvidence"
      ],

      // Date and time at which these attributes were contemporary
      "at_time": "2021-06-25T12:40:03Z",

      // Storage and integrity properties - managed by the system
      "confirmation_status": "CONFIRMED",
      "tracked": "TRACKED",
      "owner": "0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    }

Note: The Asset Identity is fixed and constant for the life of the Asset and across all stakeholders. This ensures easy traceability and assurance that all stakeholders are collaborating on the same object and see the same complete history of Events.

Deleting Assets (untracking)#

An essential value of storing evidence in DataTrails is that data is always available to stakeholders and cannot be shredded or manipulated later. Given this, it is not possible to actually delete Assets from the system, but there will be cases where it is desirable to hide Assets in the UI or omit them from default searches or compliance queries (for instance as a result of decommissioning or disposal of the corresponding physical asset).

To accommodate this need DataTrails separates the Asset estate into 2 classes: tracked Assets (those that are interesting to the system and actively recording events) and untracked Assets (those that are no longer actively interesting). When for any reason it becomes desirable to remove an Asset, the Asset owner can make it untracked so that it does not appear in lists or searches.

Caution: Untracking an Asset does not remove it or its Event history from the system; all stakeholders who previously had access to the record will continue to have access to the Event history, including the untracking event, if they look for it.

Timestamps on Events#

Lifecycle events in DataTrails give stakeholders a shared view of “Who did What When to an Asset". The “What” and the “Asset” are quite straightforward, but the “When” and “Who” can be more nuanced.

Once committed to the DataTrails system, each lifecycle Event record carries 3 separate timestamps:

• timestamp_declared - an optional user-supplied value that tells when an Event happened. This is useful for cases where the client system is off-line for a period but the user still wishes to record the accurate time and order of activities (eg inspection rounds in an air-gapped facility). If unspecified, the system sets timestamp_declared equal to timestamp_accepted (see below).
• timestamp_accepted - the time the event was actually received on the DataTrails REST interface. Set by the system, cannot be changed by the client.
• timestamp_committed - the time the event was confirmed distributed to all DLT nodes in the value chain. Set by the system, cannot be changed by the client.

Having these 3 fields enables users of DataTrails to accurately reflect what is claimed, whilst also preventing tampering and backdating of entries.

User Principals on Events#

Just as with the “When”, the “Who” of “Who Did What When to an Asset" is potentially complicated if, for example, an application or gateway is acting on behalf of some other real-world user.

Once committed to the DataTrails system, each lifecycle Event record carries 2 separate user identities:

• principal_declared - an optional user-supplied value that tells who performed an Event. This is useful for cases where the user principal/credential used to authorize the Event does not accurately or usefully reflect the real-world agent (eg a multi-user application with device-based credentials).
• principal_accepted - the actual user principal information belonging to the credential used to access the DataTrails REST interface. Set by the system and retrieved from the authorizing IDP, cannot be changed by the client.

For more detailed information on Events, and how to implement them, please refer to the Events API Reference.

Access Policies#

Sharing the right amount of information with your value chain partners is critical to creating a trustworthy shared history for Assets. It is important that every participant be able to see and contribute to the management of Assets without compromising security, commercial, or private personal information. For example, competing vendors should not see each other’s information, but both should be able to freely collaborate with their mutual customer or industry regulator.

In other scenarios, it is desirable to share basic maintenance information with a vendor or external maintenance company, whilst restricting critical operating information such as run cycles and cyber SLAs to a much smaller group.

DataTrails Access Policies are the method through which this access is defined, allowing Asset owners to collaborate with just the right partners at the right time, sharing as much or as little access to Assets as the needs of the value chain partners dictate. All transactions are private by default, meaning that only the Asset owner can see and update Asset histories until a sharing policy has been set up. This ensures ready compliance with important regimes such as GDPR and antitrust regulations, as well as allowing safe and ready collaboration with a large and diverse range of value chain partners in the DataTrails network when required.

Note: To collaborate with a value chain partner you first need to enroll them as a partner in your DataTrails Tenancy by exchanging your public DataTrails Subject Identity with each other, a little like making a new LinkedIn connection or Facebook friend.

This one-time manual process helps to underpin trust and security in your DataTrails Access Policies by ensuring that the partners represented in them are the ones you expect.

Public Attestations#

While a strict, 1-to-1 relationship might be desirable for some use cases, it is also possible that a recorded asset and associated events are recorded in a more widely accessible way. With the use of our Public setting for an asset, you can create an access policy which enables anyone to view that asset record. A viewer of that asset does not have to be registered with DataTrails, and can anonymously use our Instaproof service to check the thing they have against the public record in DataTrails.

Note: Instaproof uses the hash of a file, piece of data or digital artifact to check for associated records with that hash value. This enables users to quickly check if the thing they have is the correct, unaltered version they are expecting.

Considerations#

As with any system handling large amounts of important data, one must carefully consider the design and scope of Access Policy rules in DataTrails. Every situation is different, and the DataTrails Access Policy system is flexible and powerful enough to support most situations, but in general it is recommended to follow some basic rules:

• Aim for fewest possible number of policies: This makes it much easier to review and manage access rights.
• Balance complex, highly-specific policies with simple, broad ones: Remember rights granted by policy are additive.
• A single Access Policy can contain several permission groups, so it is possible to define a single filter to cover a particular population of Assets, then apply different rights to different sets of users and partner organizations. This is often a simpler way to manage access than to create separate Access Policies for each set of users.
• Remember attributes can change: ABAC policies are applied at time of access request, not at time of creation, so changing attributes on an asset may change which access policies apply to it. This is one of the primary advantages to an ABAC system, but needs to be kept in mind when designing access control processes.

Access Policy configuration#

DataTrails employs a principle called Attribute-Based Access Control (ABAC) for users within an organization who are given internal access to your tenant. A related concept called Organization-Based Access Control (OBAC) is provided to mediate data sharing between value chain participants who will share external access to their tenants.

Rather than applying a specific fixed policy to each Asset, or grouping them into rigid hierarchies, Access Policies are defined in terms of the observable properties (or attributes) of Assets and users, and if both match, the policy is applied. This enables much greater flexibility and expressivity than traditional hierarchical or role-based methods, whilst at the same time reducing complexity in defining sharing in large-scale systems.

DataTrails Access Policies comprise of 2 main parts:

• Filters: A list of attributes to match on Assets (this defines the scope of the policy)
• Access Permissions: A list of access rights to be granted to users on those matching Assets

A simple Access Policy may look like this:

    {
      // identity and tenant are non-modifiable system info
      "identity": "access_policies/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "tenant": "tenant/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",

      // User-friendly identifiers
      "display_name": "Sample Policy",
      "description": "An Access Policy created for DataTrails user docs"

      // Filters define which Assets this Policy applies to
      "filters": [
        {
          // Any Crate, Box, or Bag ...
          "or": [
            "attributes.arc_display_type=Crate",
            "attributes.arc_display_type=Box",
            "attributes.arc_display_type=Bag"
          ]
        },
        {
          // ... whose registered handler is either Fred or Margaret
          "or": [
            "attributes.handler=Fred",
            "attributes.handler=Margaret"
          ]
        }
      ],

      // Permissions define what can be done with the matching Assets
      "access_permissions": [
        {
          // Grant access to partner organizations by Subject Identity
          "subjects": [
            // The Subject Identity for a known partner eg ACME Co
            "subjects/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" 
          ],

          // Grant access to local users by attribute (eg email address)
          "user_attributes": [ 
            { "or": [ "email=bill@synsation.com" ] } 
          ],
          
          // Limit the APIs they can call
          "behaviours": [ "RecordEvidence" ],
          
          // Select which Asset attributes these users can see
          "asset_attributes_read": [
            "Height",
            "arc_display_name",
            "arc_display_type"
          ],

          // Select which Asset attributes these users can modify
          "asset_attributes_write": [
            "Height"
          ],

          // Select which Events from the Asset history these users can see
          "event_arc_display_type_read": [
            "Measure",
            "Open",
            "Seal"
          ],

          // Select which Events these users can contribute to the history
          "event_arc_display_type_write": [
            "Measure"
          ],

          // Note the include_attributes field is deprecated
          "include_attributes": []
        }
      ]
    }

Note: Observe that there are 2 lists in the filters which concern different attributes. The effect of this is to say that an Asset matches the filters if it matches at least one entry from every list. Or in other words, inner lists are OR, while outer lists are AND.

For example:

Filters = [
  { "or": [type=Pump, type=Valve] },
  { "or": [vendor=SynsationIndustries] },
  { "or": [location=ChicagoWest,location=ChicagoEast] }
]

In the above simplified example, the policy would apply to any Pump or Valve from SynsationIndustries installed in either the ChicagoWest or ChicagoEast sites, but it would not match:

• Other device types from SynsationIndustries;
• Pumps or Valves from any other vendor;
• SynsationIndustries Valves installed in a different location

Revoking Access#

DataTrails adopts a ‘default deny’ approach so access to an Asset Record is only possible if an Access Policy explicitly allows it.

Revoking access can therefore be achieved in a number of ways, any of which may be more or less appropriate for the circumstances:

• Remove the whole Access Policy
• Change the attributes of the Asset so that it no longer matches the Access Policy (eg change location)
• Change the attributes of the user or subject so that they no longer match the Access Policy (eg change IDP group)
• Turn off the user’s login at the IDP

Note: As with any fair decentralized system it is not possible to ‘unsee’ information. Any change in OBAC access policies including revoking OBAC access to a value chain partner only applies to new information contributed after the policy change. This ensures continued fair access to the historic evidence base for all legitimate participants whilst also maintaining control of future operations with the Asset owner.

Attachments and Blobs#

Note: DataTrails intends to be a store of metadata for your data that is independent of that data, allowing you to continue to work in the places where you already work and share data in the ways you already share data. Where it is possible, DataTrails recommends keeping your data in the places that it is already kept, rather than creating a duplicate copy in your ledger.

However, in many cases binary files, images and such can form part of that metadata trail. For such cases, DataTrails supports the uploading of Blobs which are then protected by the immutable log with the same strength and integrity as other Event attributes.

Attachments in DataTrails enable images, PDFs and other binary data to be attached to Assets and Events. This brings added richness to the evidence base and facilitates high fidelity collaboration between stakeholders.

Adding an attachment to an Event enables recording of characteristics or evidence that are very difficult to capture in the rigid structured JSON data of Attributes. For example:

• a photograph of the physical state of a device such as alignment of components or wear on tamper seals at the time of a particular inspection
• a PDF of a safety conformance report to support a maintenance event
• a software manifest to support an update
• an x-ray image

Attaching rich evidence to an Event is a two step process:

1. First a Binary Large OBject (BLOB) is uploaded
2. Then a reference to that blob is attached to the Event.

To add attachments to an Event, simply specify an attribute in the event_attributes of the POST request with a dictionary including the blob information and with "arc_attribute_type": "arc_attachment". To add or update an attachment on an Asset, put the attachment attribute in the asset_attributes of the request instead.

Note: Assets will soon be deprecated for a more flexible and powerful concept of Trails.

The change is subtle, but where Assets only allow Events to be registered against a single thing or theme, Trails will enable Events to be free of any such restriction leading to more natural expression of what happened when or who said what about what.

To prepare for this change, when writing code integrations be sure to focus on Event attributes and not mutable asset_attributes. This will ensure best performance and minimal code changes to take advantage of the newer API. Trails will still support simple properties like types, descriptions and thumbnails for search and grouping purposes.

For more detailed information on Attachments and how to implement them, please refer to the Blobs API Reference and the Attachments API Reference

The Primary Image#

Attachments on Assets and Events are named in their arc_display_name property, so that they can be searched and indexed. Names are arbitrary and may be defined according to the needs of the application, but one name is reserved and interpreted by the DataTrails services: arc_primary_image. If an Asset has an attachment attribute named arc_primary_image, then this will be used by the DataTrails web user interface and other tools as a thumbnail to represent the Asset or Event being viewed.

Note: Blobs and Attachments cannot be searched or listed as a collection in their own right: they must always be associated with an Asset or Event through an Attachment Attribute and can only be downloaded by users with appropriate access rights to that Attachment.

Note: While Attachments are generally expected to be unique, the same attachment can be applied to multiple assets, such as the case of a process manual PDF.

Geolocation#

DataTrails supports GIS coordinates on Events, which enable recording of exactly where an event took place, and when analyzed together can show the movement of an Asset (“This was scanned in London, and later sold in Manchester”):

If you’re wanting to track the movement of an Asset, or record an audit trail of where a particular Event happens, you can add arc_gis_lat and arc_gis_lng attributes to the event_attributes.

For example:

    {
      "asset_identity": "assets/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "behaviour": "RecordEvidence",
      "operation": "Record",
      "event_attributes": {
        "arc_display_type": "Sighting Report",
        "arc_description": "Observed at Fort Mason, San Francisco",
        "arc_gis_lat": "37.807338",
        "arc_gis_lng" : "-122.429286"
      }
    }

Once applied the GIS coordinates on Events are immutable.

That’s it#

These are all the basics of DataTrails. With this knowledge you can now jump straight into the API or try other topics on the DataTrails Platform.