[DD-1319] JSON streaming export @ task execution Erstellt: 15/Mär/16  Aktualisiert: 09/Nov/16  Erledigt: 04/Nov/16

Status: Fertig
Projekt: D:SWARM
Komponente(n): Keine
betrifft Version(en): Keine
Lösungsversion(en): Keine

Typ: Story Priorität: Schwer
Autor: Gängler, Thomas Bearbeiter: Nicht zugewiesen
Lösung: Fertig Stimmen: 0
Stichwörter: GDM, JSON, export, serialization, streaming, task

Verknüpfungen:
Relates
relates to DD-960 json export Aufgaben
relates to DD-1396 mark attribute paths in schema as mul... Fertig
Sprint: sprint 63, sprint 64, sprint 65, sprint 66

 Beschreibung   

maybe different flavours, e.g.

  • GDM JSON (SPO GDM statements) (media type = "application/gdm+json")
    [
      {
        "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/5f7019a6-96e3-4aae-aaac-da743e2840b9": [
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/5f7019a6-96e3-4aae-aaac-da743e2840b9"
            },
            "p": "http://www.w3.org/1999/02/22-rdf-syntax-ns#type",
            "o": {
              "uri": "http://data.slub-dresden.de/resources/1/schema#RecordType"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/5f7019a6-96e3-4aae-aaac-da743e2840b9"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#id",
            "o": {
              "v": "1234"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/5f7019a6-96e3-4aae-aaac-da743e2840b9"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#name",
            "o": {
              "v": "bla"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/5f7019a6-96e3-4aae-aaac-da743e2840b9"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#description",
            "o": {
              "v": "bla bla"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/5f7019a6-96e3-4aae-aaac-da743e2840b9"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#isbn",
            "o": {
              "v": "76587658765"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/5f7019a6-96e3-4aae-aaac-da743e2840b9"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#year",
            "o": {
              "v": "1970"
            }
          }
        ]
      },
      {
        "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/4c28b579-08d9-4dc1-9874-5fcd8a662919": [
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/4c28b579-08d9-4dc1-9874-5fcd8a662919"
            },
            "p": "http://www.w3.org/1999/02/22-rdf-syntax-ns#type",
            "o": {
              "uri": "http://data.slub-dresden.de/resources/1/schema#RecordType"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/4c28b579-08d9-4dc1-9874-5fcd8a662919"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#id",
            "o": {
              "v": "2343"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/4c28b579-08d9-4dc1-9874-5fcd8a662919"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#name",
            "o": {
              "v": "blub"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/4c28b579-08d9-4dc1-9874-5fcd8a662919"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#description",
            "o": {
              "v": "blub blub"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/4c28b579-08d9-4dc1-9874-5fcd8a662919"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#isbn",
            "o": {
              "v": "65786576576"
            }
          },
          {
            "s": {
              "uri": "http://data.slub-dresden.de/datamodels/DataModel-574990f5-4785-4020-b86a-9765bb084f16/records/4c28b579-08d9-4dc1-9874-5fcd8a662919"
            },
            "p": "http://data.slub-dresden.de/resources/1/schema#year",
            "o": {
              "v": "1960"
            }
          }
        ]
      },
    
    [...]
    
    
    ]
    
  • GDM Ordered JSON (with attribute order in entities) (GDM compact JSON; media type = "application/gdm+compact+json")
    [ {
        "http://data.slub-dresden.de/datamodels/2/records/0d801d16-cbb4-461c-b660-ac4b003d240b" : [ {
            "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" : "http://data.slub-dresden.de/resources/1/schema#RecordType"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#id" : "1234"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#name" : "bla"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#description" : "bla bla"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#isbn" : "76587658765"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#year" : "1970"
        } ]
    }, {
        "http://data.slub-dresden.de/datamodels/2/records/93b048f2-d0e9-4f71-8877-5a1615caeced" : [ {
            "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" : "http://data.slub-dresden.de/resources/1/schema#RecordType"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#id" : "2343"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#name" : "blub"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#description" : "blub blub"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#isbn" : "65786576576"
        }, {
            "http://data.slub-dresden.de/resources/1/schema#year" : "1960"
        } ]
    },
    
    [...]
    
    ]
    
  • GDM FE JSON (with attribute order in entities and separated record identifier and record content parts) (GDM Compact FE JSON; media type = "application/gdm+compact+fe+json")
    [
      {
        "__record_id": "http://data.slub-dresden.de/datamodels/DataModel-d3de7db9-ffcf-4d3b-836f-5d736fea2f55/records/030bf238-6081-4d03-8afc-b8550b0fd670",
        "__record_data": [
          {
            "http://data.slub-dresden.de/datamodels/DataModel-d3de7db9-ffcf-4d3b-836f-5d736fea2f55/schema#description": "blx blx"
          },
          {
            "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": "http://purl.org/ontology/bibo/Document"
          }
        ]
      },
      {
        "__record_id": "http://data.slub-dresden.de/datamodels/DataModel-d3de7db9-ffcf-4d3b-836f-5d736fea2f55/records/970e27de-2db8-4ea4-9428-0d959d67af4f",
        "__record_data": [
          {
            "http://data.slub-dresden.de/datamodels/DataModel-d3de7db9-ffcf-4d3b-836f-5d736fea2f55/schema#description": "blub blub"
          },
          {
            "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": "http://purl.org/ontology/bibo/Document"
          }
        ]
      },
      
     [...]
    
    ]
    
  • GDM Simple JSON (no attribute order in entities + no URIs for attribute identifier) (media type = "application/gdm+simple+json")
    [
      {
        "http://data.slub-dresden.de/datamodels/2/records/0d801d16-cbb4-461c-b660-ac4b003d240b": {
          "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": "http://data.slub-dresden.de/resources/1/schema#RecordType",
          "http://data.slub-dresden.de/resources/1/schema#id": "1234",
          "http://data.slub-dresden.de/resources/1/schema#name": "bla",
          "http://data.slub-dresden.de/resources/1/schema#description": "bla bla",
          "http://data.slub-dresden.de/resources/1/schema#isbn": "76587658765",
          "http://data.slub-dresden.de/resources/1/schema#year": "1970"
        }
      },
      {
        "http://data.slub-dresden.de/datamodels/2/records/93b048f2-d0e9-4f71-8877-5a1615caeced": {
          "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": "http://data.slub-dresden.de/resources/1/schema#RecordType",
          "http://data.slub-dresden.de/resources/1/schema#id": "2343",
          "http://data.slub-dresden.de/resources/1/schema#name": "blub",
          "http://data.slub-dresden.de/resources/1/schema#description": "blub blub",
          "http://data.slub-dresden.de/resources/1/schema#isbn": "65786576576",
          "http://data.slub-dresden.de/resources/1/schema#year": "1960"
        }
      },
    
    [...]
    
    ]
    
  • GDM Simple Short JSON (no attribute order in entities) (media type = "application/gdm+simple+short+json")
    [
      {
        "http://data.slub-dresden.de/datamodels/2/records/0d801d16-cbb4-461c-b660-ac4b003d240b": {
          "type": "http://data.slub-dresden.de/resources/1/schema#RecordType",
          "id": "1234",
          "name": "bla",
          "description": "bla bla",
          "isbn": "76587658765",
          "year": "1970"
        }
      },
      {
        "http://data.slub-dresden.de/datamodels/2/records/93b048f2-d0e9-4f71-8877-5a1615caeced": {
          "type": "http://data.slub-dresden.de/resources/1/schema#RecordType",
          "id": "2343",
          "name": "blub",
          "description": "blub blub",
          "isbn": "65786576576",
          "year": "1960"
        }
      },
    
    [...]
    
    ]
    
  • simple JSON (no attribute order in entities + no URIs for attribute identifier + no (internal) record identifiers) (media type = "application/json")
    [
      {
        "type": "http://data.slub-dresden.de/resources/1/schema#RecordType",
        "id": "1234",
        "name": "bla",
        "description": "bla bla",
        "isbn": "76587658765",
        "year": "1970"
      },
      {
        "type": "http://data.slub-dresden.de/resources/1/schema#RecordType",
        "id": "2343",
        "name": "blub",
        "description": "blub blub",
        "isbn": "65786576576",
        "year": "1960"
      },
    
    [...]
    
    ]
    
    


 Kommentare   
Kommentar durch Anonym [ 26/Mär/16 ]

Wouldn't using a JSON-LD @context allow to use CURIEs for the http://data.slub-dresden.de/resources/1/schema part of keys?

Kommentar durch Gängler, Thomas [ 29/Mär/16 ]

from my understanding (right now) the problem with compact JSON-LD is that it isn't designed for streaming. At least not in a fashion that you can simply push out your statements and serialize them as JSON-LD*. You need to know the occurring attributes (incl. their namespaces) upfront to generate a @context section (or to refer to an external one).
You (probably) can streaming process (/generate) expanded JSON-LD. Furthermore, it's also possible to streaming process (/generate) compact JSON-LD, if you know the schema (@context) upfront, which can be the case, since usually a Data Model in d:swarm refers to a Schema. So it should be possible to generate the @context section out of this piece of information (i.e. out of the Schema in the Metadata Repository).

*) you need to iterate twice over all statements - in the first iteration you can determine all occurring attributes (incl. their namespaces); so that you can generate a @context section out of it + the second iteration then serializes the statements as compact JSON-LD (or you are able to provide the Schema upfront as described above)

Kommentar durch Gängler, Thomas [ 27/Sep/16 ]

see https://github.com/zazi/dswarm-commons/tree/dd-1319

Kommentar durch Gängler, Thomas [ 04/Okt/16 ]

see https://github.com/dswarm/dswarm-commons/pull/22

Kommentar durch Gängler, Thomas [ 04/Okt/16 ]

see https://github.com/dswarm/dswarm-gdm-java/pull/9

Kommentar durch Gängler, Thomas [ 04/Okt/16 ]

see https://github.com/zazi/dswarm-backoffice-web/tree/dd-1319

Kommentar durch Gängler, Thomas [ 04/Okt/16 ]

see https://github.com/zazi/task-processing-unit-for-dswarm/tree/dd-1319

Kommentar durch Gängler, Thomas [ 04/Okt/16 ]

see https://github.com/dswarm/dswarm-documentation/wiki/JSON-Flavours

Kommentar durch Gängler, Thomas [ 18/Okt/16 ]

see https://github.com/dswarm/task-processing-unit-for-dswarm/pull/23

Kommentar durch Gängler, Thomas [ 19/Okt/16 ]

see https://github.com/dswarm/dswarm-backoffice-web/pull/70

Kommentar durch Gängler, Thomas [ 24/Okt/16 ]

see https://github.com/zazi/dswarm/tree/dd-1319

Kommentar durch Gängler, Thomas [ 25/Okt/16 ]

see https://github.com/dswarm/dswarm/pull/147

Kommentar durch Gängler, Thomas [ 26/Okt/16 ]

see https://github.com/zazi/dswarm/compare/a91a7c4409744f0e2bce2fc6098be94eb76b3b3a...0d91cd8c61693bd472ab8e6933a5479bfeb058d7 (for "application/jsc+ldj")

Kommentar durch Gängler, Thomas [ 26/Okt/16 ]

see https://github.com/zazi/dswarm/tree/dd-1319-2

Kommentar durch Gängler, Thomas [ 28/Okt/16 ]

see https://github.com/dswarm/dswarm/pull/153

Erstellt am Tue Dec 11 14:59:57 CET 2018 mit Jira 7.12.1#712002-sha1:609a50578ba6bc73dbf8b05dddd7c04a04b6807c.