Uploaded image for project: 'Data Collection, Analytics, and Events'
  1. Data Collection, Analytics, and Events
  2. DCAEGEN2-2893

VES Collector stndDefinedFields schema validation issue

XMLWordPrintable

       In the situations, where $.event.stndDefinedFields.schemaReference has direct or indirect relative-reference to other local schema file, assuming that all referenced files are in proper locations, this whole process of validation could be offline:

      $ref: 'genericNrm.yaml#/components/schemas/resources-genericNrm'

      On the other hand, I noticed that if $.event.stndDefinedFields.schemaReference has direct or indirect uri reference, validation will always try to pull that uri, regardless whether it is defined in the schema-map.json or not:

       

      $ref: 'https://forge.3gpp.org/rep/all/5G_APIs/raw/REL-16/TS29512_Npcf_SMPolicyControl.yaml#/components/schemas/RedirectInformation'
      

      It seems, that introduced public to local schema mapping mechanism, only works for the top level schema reference from $.event.stndDefinedFields.schemaReference and it does not consider references.
      The actual validation is implemented in the org.openapi4j.schema.validator.v3.SchemaValidator, which is a dependency to the dcaegen2/services/sdk project. And this is how it works. However some use cases might require full offline validation and it seems that it is not possible to achieve in the VES Collector, unless all the uri schema references (recursively) are refactored to local relative-references, and of course all the files are available locally. This would be the case for many refs from forge.3gpp.org repositories.

       

      Steps to recreate:

      1. VES Collector code base from master branch
      2. Download 3GPP OpenAPI definition and schema files to VES Collector local repo from https://forge.3gpp.org/rep/sa5/MnS/tree/Rel-16/OpenAPI (branch Rel-16, commit id 16410149a5123c9f4754d07acc2aeda80d39c511)
      3. update etc/externalRepo/schema-map.json so it reads as follows:
        [
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/faultMnS.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/faultMnS.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/heartbeatNtf.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/heartbeatNtf.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/PerDataFileReportMnS.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/PerDataFileReportMnS.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/provMnS.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/provMnS.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/Rel16/OpenAPI/faultMnS.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/Rel16/OpenAPI/faultMnS.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/Rel16/OpenAPI/heartbeatNtf.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/Rel16/OpenAPI/heartbeatNtf.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/Rel16/OpenAPI/fileDataReportingMnS.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/Rel16/OpenAPI/fileDataReportingMnS.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/Rel16/OpenAPI/perfMnS.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/Rel16/OpenAPI/perfMnS.yaml"
          },
          {
            "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/Rel16/OpenAPI/provMnS.yaml",
            "localURL": "3gpp/rep/sa5/MnS/blob/Rel16/OpenAPI/provMnS.yaml"
          }
        ]
        
      1. Complete catalog structure should look as follows (from etc/externalRepo):
      .
      ├── 3gpp
      │   └── rep
      │       └── sa5
      │           └── MnS
      │               └── blob
      │                   ├── Rel16
      │                   │   └── OpenAPI
      │                   │       ├── 5gcNrm.yaml
      │                   │       ├── comDefs.yaml
      │                   │       ├── faultMnS.yaml
      │                   │       ├── fileDataReportingMnS.yaml
      │                   │       ├── genericNrm.yaml
      │                   │       ├── heartbeatNtf.yaml
      │                   │       ├── nrNrm.yaml
      │                   │       ├── perfMnS.yaml
      │                   │       └── provMnS.yaml
      │                   └── SA88-Rel16
      │                       └── OpenAPI
      │                           ├── PerDataFileReportMnS.yaml
      │                           ├── faultMnS.yaml
      │                           ├── heartbeatNtf.yaml
      │                           └── provMnS.yaml
      └── schema-map.json
      
      1. Start VES Collector (and required DMaaP services)
      2. Send attached fm_stnd_rel16.json as VES event and use either tcpdump to catch traffic to forge.3gpp.org and/or run debugger/remote debugger attached to VES Collector process

      VES Collector log shows following error message:

       

      2021-08-26 13:27:11.684  INFO 28 [nio-8443-exec-5] o.o.d.s.s.s.e.s.m.s.ValidatorCache       : Creating new stndDefined schema validator
      2021-08-26 13:27:13.668 ERROR 28 [nio-8443-exec-5] o.o.d.r.VesRestController                : org.onap.dcaegen2.services.sdk.services.external.schema.manager.exception.IncorrectInternalFileReferenceException: Schema reference refer to existing file, but internal reference (after #) is incorrect. Failed to load document from 'https://forge.3gpp.org/rep/all/5G_APIs/raw/REL-16/TS29512_Npcf_SMPolicyControl.yaml'
      

      It complains about missing https://forge.3gpp.org/rep/all/5G_APIs/raw/REL-16/TS29512_Npcf_SMPolicyControl.yaml file. OK - it was NOT defined in the schema-map.json, but even if it was, it's irrelevant. The org.openapi4j.schema.validator.v3.SchemaValidator tries to pull it from the remote URL anyway. Interesting thing is that this file:

       

      https://forge.3gpp.org/rep/all/5G_APIs/raw/REL-16/TS29512_Npcf_SMPolicyControl.yaml has been pushed to the 3ggp repository with errors preventing this file to be properly validated as yaml file. Line 1085 contains trailing tab characters.

      Debugger show nested exception: 

      org.openapi4j.core.exception.DecodeException: Failed to decode: while scanning for the next tokenorg.openapi4j.core.exception.DecodeException: Failed to decode: while scanning for the next tokenfound character '\t(TAB)' that cannot start any token. (Do not use \t(TAB) for indentation) in 'reader', line 1085, column 18:            vplmnQos:                      ^
       at [Source: (StringReader); line: 1085, column: 17]
      

      and the above is thrown from: org.openapi4j.core.model.reference.AbstractReferenceResolver.registerDocument(AbstractReferenceResolver.java:113)

            kkuzmick kkuzmick
            mwilk mwilk
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: