Difference between revisions of "OpenAPI development guidelines"

From MECwiki
Jump to: navigation, search
 
(13 intermediate revisions by 3 users not shown)
Line 1: Line 1:
OpenAPI development guidelines for ETSI MEC ISG API specifications.
+
OpenAPI Specification (OAS) development guidelines for ETSI MEC ISG API specifications.
More general recommendations can be found at [https://forge.etsi.org/wiki/index.php/Main_Page Forge OpenAPI Guidelines].
 
  
= How to review the latest versions =
+
''(More general recommendations can be found at [https://forge.etsi.org/wiki/index.php/Main_Page Forge OpenAPI Guidelines])''
  
<big>If you are interested to access and download the raw YAML files built for each interface (currently only MEC-012 is built in this manner, the other interfaces will follow) visit [https://forge.etsi.org/jenkins/view/all/job/MEC%20-%20Multi-access%20Edge%20Computing/ latest succesful builds].</big>
+
= Latest versions =
  
<big>[[ShortGuideToReviewInGerrit|A short guide to review in Gerrit]]</big>
+
Latest versions of the OAS representations are available in their respective repositories ([https://forge.etsi.org/rep/mec here])
  
If you are willing to contribute or if you want more information, contact [mailto:cti_support@etsi.org?subject=MEC%20openapis%20development: ETSI CTI]. ETSI CTI have also made a [https://www.youtube.com/watch?v=YXO0R61Dcg8 YouTube] tutorial available, which although based on NFV rather than MEC provides a valuable overall guide.  
+
Latest CI test results of the OAS representations are available [https://forge.etsi.org/jenkins/view/all/job/MEC%20-%20Multi-access%20Edge%20Computing/ here]
 +
 
 +
= Contributing =
 +
 
 +
If you are willing to contribute or if you want more information, contact [mailto:cti_support@etsi.org?subject=MEC%20openapis%20development: ETSI CTI].  
 +
 
 +
ETSI CTI has also made a [https://www.youtube.com/watch?v=YXO0R61Dcg8 YouTube] tutorial available (based on NFV) that provides guidance on how to contribute for MEC.
  
 
= API development =
 
= API development =
Line 14: Line 19:
 
== General guidelines ==
 
== General guidelines ==
  
MEC APIs are currently described using the [https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md OpenAPI 2.0 specification]. Members of the ETSI MEC ISG can view a tutorial [https://docbox.etsi.org/ISG/MEC/05-CONTRIBUTIONS/2017//MEC(17)000541_MEC023_Creating_a_OpenAPI_spec_compliant_API_description.pptx presentation]. That can be made available outside the ISG on request.
+
As of January 2021, OAS representations have been updated to reflect the latest released versions of their corresponding GS (e.g. v2.1.x);  the representations follow [https://swagger.io/specification/ OpenAPI 3.0 specification] language.  
 +
 
 +
The following figure provides a high level overview of the OAS creation and usage.
 +
[[File:ToolchainOverview.png|center|thumb|500x500px| class=img-responsive]]
 +
 
 +
As shown on the figure, OAS YAML file is used in 3 different contexts:
 +
# To accurately reflect the GS API as released by ETSI MEC ISG
 +
# To provide graphical UI documentation as an input to SwaggerUI JS Application
 +
# To provide out-of-the-box API client library/package or server stub as an input to code generation tools
 +
 
 +
As such, when developing or updating the OAS YAML representations, the 3 usage described above should be considered.
 +
 
 +
The following sections provide guidelines on how to develop and validate OAS represnetaions.
 +
 
 +
== OAS YAML Generation ==
 +
Initial OAS generation involves using the ETSI maintained doc2oas tool; the figure below shows the steps involved in using this tool.
 +
 
 +
The following figure outlines doc2oas usage.
 +
[[File:Doc2oasToolchain.png|center|thumb]]
 +
 
 +
Step 1 - Create <code>MEC-XYZ_config.yaml</code> file
 +
* this file contains all external types not defined in the GS in (OAS language)
 +
 
 +
* existing config files can be used as a template, types can be re-used from existing config files
 +
* to quickly identify external types, Step2 can be performed with an empty config file - the resulting yaml file will show all unresolved types in the Swagger Editor
 +
 
 +
Step 2 - Use the <code>doc2oas.sh</code> script to generate <code>MEC-XYZ.yaml</code>
 +
* the new specification must first be added to <code>doc2oas.sh</code> script
 +
 
 +
Step 3 - Add endpoints as per the GS specification
 +
* follow the model from existing specifications
 +
* the end result should not show any error in Swagger Editor
 +
 
 +
== OAS YAML Validation ==
 +
Once the YAML file is available, the following validation steps should be performed to verify the validity in the different usage contexts
 +
 
 +
=== Level 1 Validation - OAS syntax ===
 +
{| class="wikitable"
 +
|+
 +
!Validation Tool
 +
!Validation Criteria
 +
|-
 +
|Swagger-Editor
 +
|<nowiki>- No remaining warning or errors when opening OAS YAML</nowiki>
 +
|-
 +
|Swagger-CLI
 +
| - Part of CI validation done via Jenkins in the repository
 +
- Swagger-CLI reports no error/warnings
 +
 
 +
- <code>swagger-cli validate <oas-file.yaml></code>
 +
|-
 +
|Speccy
 +
| - Part of CI validation done via Jenkins in the repository
 +
- Speccy reports no error/warnings
 +
 
 +
- <code>speccy lint "<oas-file.yaml>"</code>
 +
|}
 +
 
 +
=== Level 2 Validation - Swagger UI ===
 +
{| class="wikitable"
 +
!Validation Tool
 +
!Validation Criteria
 +
|-
 +
|Swagger-UI
 +
|<nowiki>- No error when viewing the file via Swagger UI</nowiki>
 +
 
 +
- Thorough visual inspection of endpoints and data model
 +
|}
 +
 
 +
=== Level 3 Validation - Code Generation ===
 +
{| class="wikitable"
 +
!Validation Tool
 +
!Validation Criteria
 +
|-
 +
|swagger-codegen-cli-v3
 +
|<nowiki>- From CLI tool: generate client API library/package and server stub for Java and Golang</nowiki>
 +
 
 +
- Code generation succeeds
 +
 
 +
- Extra: execute server stub without crash
 +
 
 +
- Example Golang Server & Client
 +
 
 +
<code>java -jar swagger-codegen-cli-v3.jar generate -i <oas-file.yaml> -l go-server -o ./server-gen -DpackageName=server</code>
 +
 
 +
<code>java -jar swagger-codegen-cli-v3.jar generate -i <oas-file.yaml> -l go -o ./client-gen -DpackageName=client</code>
 +
|}
  
A migration to the latest version of the [https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md OpenAPI 3.0 specification] is being actively pursued within the ISG. However, it is desirable that auto-generation of client and server stubs are supported through [https://github.com/swagger-api/swagger-codegen swagger-codegen], which is still in development (reference 3.0.0_enhancements branch).
+
== Proto3 Generation & Validation ==
 +
Some API representations include a Protobuf representation along with the OAS3 representation; this representation is generated using the tool Open API Generator
  
=== Root API files ===
+
=== Proto3 Generation ===
 +
{| class="wikitable"
 +
!Generation Tool
 +
!Generation Criteria
 +
|-
 +
|openapi-generator-cli-5.0.0-beta2
 +
|<nowiki>- From CLI: generate Proto3 representation</nowiki>
  
Since the specification requires the <code>apiVersion</code> be after <code>apiName</code> in the resource path, each interface will have a separate OpenAPI root file.
+
- Proto3 generation succeeds
  
In details, paths are required to start with:
+
- <code>java -jar openapi-generator-cli-5.0.0-beta2.jar generate -i <oas-file.yaml> -g protobuf-schema -o proto/ --package-name <mec1234></code>
 +
 
 +
- Full inspection of the generated Proto3 is mandatory
 +
 
 +
- At the time of writing, some Proto3 generated files needs to be touched manually: enumerations, structures containing allOf, oneOf, anyOf
 +
|}
 +
 
 +
=== Proto3 Validation ===
 +
{| class="wikitable"
 +
!Validation Tool
 +
!Validation Criteria
 +
|-
 +
|protoc
 +
|<nowiki>- From CLI: generate source code from Proto3 representation</nowiki>
 +
 
 +
- Code generation succeeds
 +
 
 +
- Sometimes, when generating code, it may be necessary to include the file [https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/empty.proto empty.proto]
 +
 
 +
Example generate Go and Java
 
<pre>
 
<pre>
<apiRoot>/<apiName>/<apiVersion>
+
mkdir go java
</pre>
 
  
the only way to model this with OAS is using separate files for each interface. This will also help readability and clarity.
+
protoc --go_out=go --java_out=java services/*
  
=== Definitions ===
+
protoc --go_out=go --java_out=java models/*
Definitions are currently not shared across API descriptions and there is no cross referencing. This is also the case in the group specifications themselves.
+
</pre>
 
+
|}
=== Resources ===
 
It's encouraged to create a separate file for each resource (as is the case for Mp1 and Location), although only the complete description file is provided for several APIs.
 
  
 
== Branching strategy ==
 
== Branching strategy ==
 +
Development follows a simplified version of the popular [https://nvie.com/posts/a-successful-git-branching-model/ GitFlow]
 +
* <code>master</code> - released version of the OAS representation
 +
* <code>develop</code> - final versions of OAS representations development, before they are approved to be merged  on master
 +
* <code>feature branch</code> - work branch, the final result should be merged on develop
  
* Within the repository there will be a branch for each ''active'' version of the interfaces. A version is ''active'' if it is still under maintenance in the WG.
+
For example, work performed by a Specialist Task Force (stf123)
* Git tags will be used to label a commit on a branch with a specific revision of the deliberable.
+
# STF - From <code>develop</code>, create a <code>stf123</code> <code>feature branch</code>
 +
# STF - Once work is completed & validation pass, merge the <code>stf123</code> to <code>develop</code>
 +
# STF - Create a Merge Request from <code>develop</code> to <code>master</code>
 +
# SC - A member from the STF Steering Committee will perform the final merge
 +
# SC - Apply a label corresponding the the GS version
  
 
== Folder structure and file names ==
 
== Folder structure and file names ==
  
 
=== Repository structure ===
 
=== Repository structure ===
 
 
<pre>
 
<pre>
 
repository_root                    i.e. MEC
 
repository_root                    i.e. MEC
|- Group Specification (GS) x      e.g. MEC.GS-011
+
├── LICENSE
  |- Readme.md                     General API information
+
├── proto3                          Protobuf representation generated from OAS yaml
  |- APIx.json                     Auto-generated JSON
+
│   ├── models                      Protobuf models
  |- APIx.yaml                    Main OAS file (auto-generated using the APIx.split.yaml file: multi-file-swagger Mp1.split.yaml > Mp1.json), e.g. Mp1.yaml
+
│   ├── README.md                  Protobuf general info
  |- APIx.split.yaml               Definition file split into its component parts, to bring it together use [https://github.com/mohsen1/multi-file-swagger-example multi-file-swagger]
+
│   └── services                    Protobuf services
      |- definitions
+
├── README.md                       General API information
        | - DataType1.yaml        e.g. CurrentTime.yaml
+
├── MEC-XYZ.json                   Auto-generated JSON (From Swagger Editor -> Save As json)
        | - DataType2.yaml
+
└── MEC-XYZ.yaml                   Definition flat file used by Swagger-UI, Swagger-Editor, Swagger-CodeGen, etc.
      |- examples                  Example responses
+
</pre>''NOTE: Naming of of the JSON and YAML files is generally chosen by the DECODE group and does not necessarily follow the pattern outlined above.''
        | - Example1.yaml          e.g. ServiceInfo.yaml
 
        | - Example2.yaml
 
      |- externalDocs              Links to external docs, i.e. the GS
 
        | - index.yaml
 
      |- info                      General info regarding API, e.g. Copyright
 
        | - index.yaml
 
      |- parameters                API Body/Query/Path Parameters
 
        | - Parameter1.yaml        e.g. Body.DnsRule.yaml
 
        | - Parameter2.yaml        e.g. Path.DnsRuleId.yaml
 
      |- paths                      Paths associated with the API
 
        | - PathA.yaml            High level path, e.g. Services.yaml
 
        | - PathA.{method}.yaml    Lower level path, e.g. Services.GET.yaml
 
|- Group Specification y
 
  |- ...
 
|- Group Specification z
 
  |- ...
 
|- Readme.md
 
|- LICENSE
 
</pre>
 
 
 
=== File names ===
 
 
 
The naming schema provided in the examples for the "Repository structure" above is followed.
 

Latest revision as of 09:47, 1 December 2021

OpenAPI Specification (OAS) development guidelines for ETSI MEC ISG API specifications.

(More general recommendations can be found at Forge OpenAPI Guidelines)

Latest versions

Latest versions of the OAS representations are available in their respective repositories (here)

Latest CI test results of the OAS representations are available here

Contributing

If you are willing to contribute or if you want more information, contact ETSI CTI.

ETSI CTI has also made a YouTube tutorial available (based on NFV) that provides guidance on how to contribute for MEC.

API development

General guidelines

As of January 2021, OAS representations have been updated to reflect the latest released versions of their corresponding GS (e.g. v2.1.x); the representations follow OpenAPI 3.0 specification language.

The following figure provides a high level overview of the OAS creation and usage.

ToolchainOverview.png

As shown on the figure, OAS YAML file is used in 3 different contexts:

  1. To accurately reflect the GS API as released by ETSI MEC ISG
  2. To provide graphical UI documentation as an input to SwaggerUI JS Application
  3. To provide out-of-the-box API client library/package or server stub as an input to code generation tools

As such, when developing or updating the OAS YAML representations, the 3 usage described above should be considered.

The following sections provide guidelines on how to develop and validate OAS represnetaions.

OAS YAML Generation

Initial OAS generation involves using the ETSI maintained doc2oas tool; the figure below shows the steps involved in using this tool.

The following figure outlines doc2oas usage.

Doc2oasToolchain.png

Step 1 - Create MEC-XYZ_config.yaml file

  • this file contains all external types not defined in the GS in (OAS language)
  • existing config files can be used as a template, types can be re-used from existing config files
  • to quickly identify external types, Step2 can be performed with an empty config file - the resulting yaml file will show all unresolved types in the Swagger Editor

Step 2 - Use the doc2oas.sh script to generate MEC-XYZ.yaml

  • the new specification must first be added to doc2oas.sh script

Step 3 - Add endpoints as per the GS specification

  • follow the model from existing specifications
  • the end result should not show any error in Swagger Editor

OAS YAML Validation

Once the YAML file is available, the following validation steps should be performed to verify the validity in the different usage contexts

Level 1 Validation - OAS syntax

Validation Tool Validation Criteria
Swagger-Editor - No remaining warning or errors when opening OAS YAML
Swagger-CLI - Part of CI validation done via Jenkins in the repository

- Swagger-CLI reports no error/warnings

- swagger-cli validate <oas-file.yaml>

Speccy - Part of CI validation done via Jenkins in the repository

- Speccy reports no error/warnings

- speccy lint "<oas-file.yaml>"

Level 2 Validation - Swagger UI

Validation Tool Validation Criteria
Swagger-UI - No error when viewing the file via Swagger UI

- Thorough visual inspection of endpoints and data model

Level 3 Validation - Code Generation

Validation Tool Validation Criteria
swagger-codegen-cli-v3 - From CLI tool: generate client API library/package and server stub for Java and Golang

- Code generation succeeds

- Extra: execute server stub without crash

- Example Golang Server & Client

java -jar swagger-codegen-cli-v3.jar generate -i <oas-file.yaml> -l go-server -o ./server-gen -DpackageName=server

java -jar swagger-codegen-cli-v3.jar generate -i <oas-file.yaml> -l go -o ./client-gen -DpackageName=client

Proto3 Generation & Validation

Some API representations include a Protobuf representation along with the OAS3 representation; this representation is generated using the tool Open API Generator

Proto3 Generation

Generation Tool Generation Criteria
openapi-generator-cli-5.0.0-beta2 - From CLI: generate Proto3 representation

- Proto3 generation succeeds

- java -jar openapi-generator-cli-5.0.0-beta2.jar generate -i <oas-file.yaml> -g protobuf-schema -o proto/ --package-name <mec1234>

- Full inspection of the generated Proto3 is mandatory

- At the time of writing, some Proto3 generated files needs to be touched manually: enumerations, structures containing allOf, oneOf, anyOf

Proto3 Validation

Validation Tool Validation Criteria
protoc - From CLI: generate source code from Proto3 representation

- Code generation succeeds

- Sometimes, when generating code, it may be necessary to include the file empty.proto

Example generate Go and Java

mkdir go java

protoc --go_out=go --java_out=java services/*

protoc --go_out=go --java_out=java models/*

Branching strategy

Development follows a simplified version of the popular GitFlow

  • master - released version of the OAS representation
  • develop - final versions of OAS representations development, before they are approved to be merged on master
  • feature branch - work branch, the final result should be merged on develop

For example, work performed by a Specialist Task Force (stf123)

  1. STF - From develop, create a stf123 feature branch
  2. STF - Once work is completed & validation pass, merge the stf123 to develop
  3. STF - Create a Merge Request from develop to master
  4. SC - A member from the STF Steering Committee will perform the final merge
  5. SC - Apply a label corresponding the the GS version

Folder structure and file names

Repository structure

repository_root                     i.e. MEC
├── LICENSE
├── proto3                          Protobuf representation generated from OAS yaml
│   ├── models                      Protobuf models
│   ├── README.md                   Protobuf general info
│   └── services                    Protobuf services
├── README.md                       General API information
├── MEC-XYZ.json                    Auto-generated JSON (From Swagger Editor -> Save As json)
└── MEC-XYZ.yaml                    Definition flat file used by Swagger-UI, Swagger-Editor, Swagger-CodeGen, etc.

NOTE: Naming of of the JSON and YAML files is generally chosen by the DECODE group and does not necessarily follow the pattern outlined above.