SDK Development

One of the key objectives we had when building the first group of SDKs was to enable developers to change quickly between programming languages without having to adapt to a completely different design. This document intends to guide developers to ship Symbol-based SDKs that share the same design to achieve interoperability.

Architecture

Package Organization

../_images/sdk-architecture.png

Package organization diagram

Infrastructure

This package includes the generated API client and DTOs. The HTTP requests are made following the Repository Pattern, and they return Symbol domain immutable models via the Observable Pattern.

Models

The Symbol domain models are immutable by definition, meaning that the developer cannot change their attributes. Instead, the developers have to create new Transactions and dispatch them to the blockchain via TransactionHTTP.

Services

Common operations that require combining multiple REST API requests.

Characteristics

  • Standardized Contracts: Guaranteeing interoperability and harmonization of data models.
  • Loose Coupling: Reducing the degree of component coupling fosters.
  • Abstraction: Increasing the long-term consistency of interoperability and allowing underlying components to evolve independently.
  • Reusability: Enabling high-level interoperability between modules and potential component consumers.
  • Stateless: Increasing availability and scalability of components allowing them to interoperate more frequently and reliably.
  • Composability: For components to be effectively composable, they must be interoperable.

Reactive

The Symbol SDK uses the ReactiveX Library intensely.

The benefits of using a reactive approach are:

  • Functional: Developers can avoid complex stateful programs using clean input/output functions over observable streams.
  • Less is more: ReactiveX’s operators often reduce what was once an elaborate challenge into a few lines of code.
  • Async error handling: Traditional try/catch is weak for error handling in asynchronous computations, but ReactiveX will offer developers the proper tools to handle errors.
  • Concurrency: Observables and Schedulers in ReactiveX allow the programmer to abstract away low-level threading, synchronization, and concurrency issues.
  • Frontend: Manipulation of UI events and API responses on the Web using RxJS.
  • Backend: Embrace ReactiveX’s asynchronicity, enabling concurrency and implementation independence.

If you are new to Reactive Programming, we encourage you to start with the online guide Learn RxJS.

Before starting

  1. Review the technical documentation to become familiar with the Symbol built-in features.
  2. Setup the Symbol local environment via docker.
  3. Check the API reference and play with offered set of endpoints.
  4. Become familiar with the current SDK via code examples and CLI .
  5. Join our Slack to ask Symbol related questions.
  6. Be sure no one is already working on the SDK you want to create. Check the repository list and comment on your intentions in #sig-api channel.
  7. Claim the SDK forking this repository and add a new entry to the repository list.

Creating the project

You can base your work on the TypeScript SDK. The TypeScript version is the first SDK getting the latest updates. Check regularly the Changelog to be sure you didn’t miss any code change update.

Create a new repository, preferably on GitHub, with:

  1. The README with the instructions to install the SDK.
  2. The Code of Conduct.
  3. The Contributors guidelines to help others know how they can help you.

Testing

A project with good test coverage it’s more likely to be used and trusted by the developers!

We strongly suggest to do Test-Driven Development or Unit-Testing (test last). If you need inspiration, feel free to adapt directly the same tests we did.

Once you have written some tests, setup a Continuous Integration (CI) system to run the test suite and code linter automatically. We use travis-ci, but feel free to use the one that suits you best.

Also, we strive to keep our codebases with a unit test coverage of 80% or higher. We use coveralls to monitor test coverage.

Infrastucture

The OpenAPI Generator handles the API and DTOs generation. It supports multiple languages, and hopefully, yours is on the list.

These are the steps we followed to generate the Typescript DTOs (data transfer objects):

  1. Download the latest Symbol OpenAPI spec from GitHub releases.
  2. Install the OpenApi generator CLI.
  1. Generate the DTOs for the programming language selected.
openapi-generator generate -i ./openapi3.yml -g typescript-node -o ./symbol-ts-sdk/ && rm -R symbol-ts-sdk/test
  1. The generated lib is normally published into a central repository (e.g. maven, npm). The SDKs depend on those libraries like any other third party dependency. To automate the deployment of the packages, including the generator for the selected programming language in the symbol-openapi-generator project.
  2. Drop the generated client classes and implement them using the Repository pattern returning Observables of ReactiveX.

Note

The SDK for TypeScript currently chooses the typescript-node template from the OpenAPI Generator, but there are also other templates available to fit for other purposes. The SDK has interfaced out all the Http Repositories so that different implementations can be applied.

Example of repositories and implementations:

See the complete list of repositories and implementations.

  1. The repositories return models instead of DTOs. You will need to code the models before finishing the API wrapper.

Models

By default, models are immutable and aim to hide the complexity, like type conversion or relationship between objects.

Example of models implementation:

See the complete list of models.

You will find in the implementations different invariants to ensure the object is well constructed and a nicer API is published.

Particular decisions we considered:

  • UInt64 support: While Java supports big numbers, for example, JavaScript doesn’t. The JavaScript SDK has a custom class to handle the uint64 types. If your language supports uint64, use that implementation instead.
  • API conversions: Sometimes, the data returned by API is compressed. You might need to convert those types for the user.
  • Namespace id: At creation time you add the string name, but when you receive the Namespace from the network, it comes in formatted as uint64 id. A specific endpoint returns the Namespace string name.

Transaction Serialization

The catbuffer library defines the protocol to serialize and deserialize Symbol entities.

In combination with the catbuffer-generators project, developers can generate builder classes for a given set of programming languages. For example, the Symbol SDK uses the generated code to operate with the entities in binary form.

Note

If there is no generator for the programming language selected, you will need to develop it first. You can base your work on the generator for TypeScript.

If there is a generator, follow the next steps to generate the builders for all the existent entities:

  1. Clone the catbuffer-generators repository recursively.
git clone --recursive git@github.com:nemtech/catbuffer-generators.git
  1. Install the package requirements.
pip install -r requirements.txt
  1. Clone the catbuffer repository inside the catbuffer-generators folder.
  2. Generate code for all the schemas running the following command under the catbuffer-generators directory, replacing cpp_builder for the targeted programming language.
python scripts/generate_all.sh cpp_builder

The previous command creates a new file for every schema under the catbuffer/_generated/cpp_builder folder.

  1. Publish the generated code into a central repository (e.g. Maven, NPM) and make the SDK dependant on this library. For every transaction type, use the generated builders to serialize and deserialize transactions.

Here you can find some examples of how we used transactions builders:

See the complete list of transactions.

KeyPair and Cryptographic functions

Note

This section is incomplete.

Cryptographic functions are required to sign transactions. All the crypto-related functions can be found under the core/crypto module.

SDKs use standard tweetnacl (ed2559) for key pair generation, address derivation (from public key) and signings:

  • Keypairs are based on tweetnacl 64 bytes secretKey (public + private) using SHA-512.
  • Signatures use tweetnacl detached mode and also get generated using SHA-512.

Finally, pay special attention to the test vectors. The best way to make sure your implementation is correct is to use the test vectors files as inputs and expected outputs.

Examples of vector tests:

Services

Services combine multiple REST API requests and provide developers with handy methods that cannot be retrieved directly from the API.

Services are considered “nice to have” features, and these usually are not required to consider the SDK complete. We recommend starting coding services only if you have a fully operational and well-tested SDK first.

Examples of services:

  • AggregateTransactionService: Helps application developers to announce aggregate transactions without having to develop the logic to wait for the hash lock confirmation.
  • MetadataTransactionService: Creates metadata transactions without having to pass the previous value.
  • BlockService: Provides with methods to verify that the data returned by a given node is valid.

See the complete list of services.

Documenting the SDK

The SDKs need to be adopted by other developers. As the main developer, no one knows better than you how the SDK works. Consider helping others and spread the usage of the SDK by providing the following documentation.

Publishing the SDK as official

To make an SDK officially supported, submit it as a NIP. The reason behind the Symbol Improvement Proposal is to ensure that the new libraries are reviewed, tested, and shared among Symbol developers.