WSTP support 🔗

LibraryLink allows a LinkObject to be passed as an argument which may then exchange data between your library and the Kernel using Wolfram Symbolic Transfer Protocol (WSTP, also known as MathLink). The original WSTP is a C style API with error codes, macros, manual memory management, etc. LLU provides a wrapper for the LinkObject called WSStream.

WSStream is actually a class template in the namespace LLU parameterized by the default encodings to be used for strings, but for the sake of clarity, both the template parameters and the namespace are skipped in the remainder of this text.

Main features 🔗

Convenient syntax 🔗

In LLU WSTP is interpreted as an I/O stream, so operators << and >> are utilized to make the syntax cleaner and more concise. This frees developers from the responsibility to choose the proper WSTP API function for the data they intend to read or write.

Error checking 🔗

Each call to WSTP API has its return status checked. An exception is thrown on failures which carries some debug info to help locate the problem. Sample debug info looks like this:

Error code reported by WSTP: 48
"Unable to convert from given character encoding to WSTP encoding"
Additional debug info: WSPutUTF8String

Memory cleanup 🔗

WSRelease* no longer needs to be called on the data received from WSTP. The LLU framework does it for you.

Automated handling of common data types 🔗

Some sophisticated types can be sent to Wolfram Language directly via a WSStream class. For example nested maps:

std::map<std::string, std::map<int, std::vector<double>>> myNestedMap

Just write ms << myNestedMap and a nested Association will be returned. It works in the other direction too. Obviously, for the above to work, the key and value types in the map must be supported by WSStream (i.e. there must exist an overload of WSStream::operator<< that takes an argument of given type).

User-defined classes 🔗

Suppose you have a structure

struct Color {
    double red;
    double green;
    double blue;
};

It is enough to overload operator<< like this:

1
2
3
    WSStream& operator<<(WSStream& ms, const Color& c) {
        return ms << WS::Function("RGBColor", 3) << c.red << c.green << c.blue;
    }

Objects of class Color can now be sent directly via WSStream.

Example 🔗

Let’s compare the same piece of code written in plain LibraryLink with one written with LLU and WSStream. Here is the plain LibraryLink code:

    if (!WSNewPacket(mlp)) {
        wsErr = -1;
        goto cleanup;
    }
    if (!WSPutFunction(mlp, "List", nframes)) {
        wsErr = -1;
        goto cleanup;
    }
    for (auto& f : extractedFrames) {
        if (!WSPutFunction(mlp, "List", 7)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutFunction(mlp, "Rule", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutString(mlp, "ImageSize")) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutFunction(mlp, "List", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->width)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->height)) {
            wsErr = -1;
            goto cleanup;
        }
        // ...
        if (!WSPutFunction(mlp, "Rule", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutString(mlp, "ImageOffset")) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutFunction(mlp, "List", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->left)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->top)) {
            wsErr = -1;
            goto cleanup;
        }
        // ...
        if (!WSPutFunction(mlp, "Rule", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutString(mlp, "UserInputFlag")) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutSymbol(mlp, f->userInputFlag == true ? "True" : "False")) {
            wsErr = -1;
            goto cleanup;
        }
    }
    if (!WSEndPacket(mlp)) {
            /* unable to send the end-of-packet sequence to mlp */
    }
    if (!WSFlush(mlp)){
            /* unable to flush any buffered output data in mlp */
    }

and now the same code using WSStream:

    WSStream ms(mlp);

    ms << WS::NewPacket;
    ms << WS::List(nframes);

    for (auto& f : extractedFrames) {
        ms << WS::List(7)
            << WS::Rule
                << "ImageSize"
                << WS::List(2) << f->width << f->height
            // ...
            << WS::Rule
                << "ImageOffset"
                << WS::List(2) << f->left << f->top
            // ...
            << WS::Rule
                << "UserInputFlag"
                << f->userInputFlag
    }

    ms << WS::EndPacket << WS::Flush;

Expressions of unknown length 🔗

Whenever you send an expression via WSTP you have to first specify the head and the number of arguments. This is not very flexible for example when an unknown number of contents are being read from a file.

As a workaround, one can create a temporary loopback link, accumulate all the arguments there (without the head), count the arguments, and then send everything to the “main” link as usual.

The same strategy has been incorporated into WSStream so that developers do not have to implement it. Now you can send a List like this:

1
2
3
4
5
6
7
    WSStream ms(mlp);

    ms << WS::BeginExpr("List");
    while (dataFromFile != EOF) {
            // process data from file and send to WSStream
    }
    ms << WS::EndExpr();

Warning

This feature should only be used if necessary since it requires a temporary link and makes extra copies of data. Simple benchmarks showed a ~2x slowdown compared to the usual WSPutFunction.

API reference 🔗

class LLU::WSStream 🔗

Wrapper class over WSTP with a stream-like interface.

WSStream resides in LLU namespace, whereas other WSTP-related classes can be found in LLU::WS namespace.

Template Parameters

  • EncodingIn: - default encoding to use when reading strings from WSTP

  • EncodingOut: - default encoding to use when writing strings to WSTP

Public Types

using StreamToken = WSStream &(*)(WSStream&) 🔗

Type of elements that can be sent via WSTP with no arguments, for example WS::Flush.

using BidirStreamToken = WSStream &(*)(WSStream&, WS::Direction) 🔗

Type of elements that can be either sent or received via WSTP with no arguments, for example WS::Rule.

using LoopbackData = std::pair<std::string, WSLINK> 🔗

Type of data stored on the stack to facilitate sending expressions of a priori unknown length.

Public Functions

WSStream(WSLINK mlp) 🔗

Constructs new WSStream.

Parameters

  • [in] mlp: - low-level object of type WSLINK received from LibraryLink

WSStream(WSLINK mlp, int argc) 🔗

Constructs new WSStream and checks whether there is a list of argc arguments on the LinkObject waiting to be read.

Parameters

  • [in] mlp: - low-level object of type WSLINK received from LibraryLink

  • [in] argc: - expected number of arguments

WSStream(WSLINK mlp, const std::string &head, int argc) 🔗

Constructs new WSStream and checks whether there is a function with head head and argc arguments on the LinkObject waiting to be read.

Parameters

  • [in] mlp: - low-level object of type WSLINK received from LibraryLink\

  • [in] head: - expected head of expression on the Link

  • [in] argc: - expected number of arguments

Exceptions

  • see: WSStream::testHead(const std::string&, int);

Note

arguments passed to the library function will almost always be wrapped in a List, so if not sure pass “List” as head

~WSStream() = default 🔗

Default destructor.

WSLINK &get() noexcept 🔗

Returns a reference to underlying low-level WSTP handle.

template<typename Iterator, typename = enable_if_input_iterator<Iterator>>
void sendRange(Iterator begin, Iterator end) 🔗

Sends any range as List.

Template Parameters

  • InputIterator: - type that is an iterator

Parameters

  • [in] begin: - iterator to the first element of the range

  • [in] end: - iterator past the last element of the range

template<typename Iterator, typename = enable_if_input_iterator<Iterator>>
void sendRange(Iterator begin, Iterator end, const std::string &head) 🔗

Sends a range of elements as top-level expression with arbitrary head.

Template Parameters

  • InputIterator: - type that is an iterator

Parameters

  • [in] begin: - iterator to the first element of the range

  • [in] end: - iterator past the last element of the range

  • [in] head: - head of the top-level expression

WSStream &operator<<(StreamToken f) 🔗

Sends a stream token via WSTP.

Parameters

  • [in] f: - a stream token, i.e. an element that can be sent via WSTP with no arguments, for example WS::Flush

WSStream &operator<<(BidirStreamToken f) 🔗

Sends a bidirectional stream token via WSTP.

Parameters

  • [in] f: - an element that can be either sent or received via WSTP with no arguments, for example WS::Rule

WSStream &operator<<(const WS::Symbol &s) 🔗

Sends a top-level symbol via WSTP.

Parameters

  • [in] s: - a symbol

Exceptions

  • ErrorName::WSPutSymbolError:

See

WS::Symbol

WSStream &operator<<(const WS::Function &f) 🔗

Sends a top-level function via WSTP, function arguments should be sent immediately after.

Parameters

  • [in] f: - a function

Exceptions

  • ErrorName::WSPutFunctionError:

See

WS::Function

WSStream &operator<<(const WS::Missing &f) 🔗

Sends a top-level expression of the form Missing[“reason”].

Parameters

  • [in] f: - WS::Missing object with a reason

Exceptions

  • ErrorName::WSPutFunctionError:

See

WS::Missing

WSStream &operator<<(const WS::BeginExpr &expr) 🔗

Starts sending a new expression where the number of arguments is not known a priori.

Parameters

  • [in] expr: - object of class BeginExpr that stores expression head as string

WSStream &operator<<(const WS::DropExpr &expr) 🔗

Drops current expression that was initiated with BeginExpr.

Parameters

  • [in] expr: - object of class DropExpr

WSStream &operator<<(const WS::EndExpr &expr) 🔗

Ends current expression that was initiated with BeginExpr, prepends the head from BeginExpr and sends everything to the “parent” link.

Parameters

  • [in] expr: - object of class EndExpr

WSStream &operator<<(bool b) 🔗

Sends a boolean value via WSTP, it is translated to True or False in Mathematica.

Parameters

  • [in] b: - a boolean value

Exceptions

  • ErrorName::WSPutSymbolError:

WSStream &operator<<(mint i) 🔗

Sends a mint value.

Parameters

  • [in] i: - a mint value

Exceptions

  • ErrorName::WSPutScalarError:

template<typename T>
WSStream &operator<<(const WS::ArrayData<T> &a) 🔗

Sends a WSTP array.

Template Parameters

  • T: - array element type

Parameters

  • [in] a: - ArrayData to be sent

Exceptions

  • ErrorName::WSPutArrayError:

See

WS::ArrayData<T>

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingMultidimensionalArrays.html

template<typename T>
WSStream &operator<<(const WS::ListData<T> &l) 🔗

Sends a WSTP list.

Template Parameters

  • T: - list element type

Parameters

  • [in] l: - ListData to be sent

Exceptions

  • ErrorName::WSPutListError:

See

WS::ListData<T>

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingLists.html

template<typename T, typename D>
WSStream &operator<<(const std::unique_ptr<T, D> &p) 🔗

Sends an object owned by unique pointer.

Template Parameters

  • T: - list element type

  • D: - destructor type, not really relevant

Parameters

  • [in] p: - pointer to the object to be sent

template<typename T>
WSStream &operator<<(const std::vector<T> &l) 🔗

Sends a std::vector via WSTP, it is interpreted as a List in Mathematica.

Template Parameters

  • T: - vector element type (types supported in WSPut*List will be handled more efficiently)

Parameters

  • [in] l: - std::vector to be sent

Exceptions

  • ErrorName::WSPutListError:

template<WS::Encoding E>
WSStream &operator<<(const WS::StringData<E> &s) 🔗

Sends a WSTP string.

Template Parameters

  • E: - encoding of the string (it determines which function from WSPut*String family to use)

Parameters

  • [in] s: - WS::StringData to be sent

Exceptions

  • ErrorName::WSPutStringError:

See

WS::StringData<E>

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingStrings.html

template<WS::Encoding E, typename T>
WSStream &operator<<(const WS::PutAs<E, T> &wrp) 🔗

Sends all strings within a given object using specified character encoding.

Normally, when you send a string WSStream chooses the appropriate WSTP function based on the EncodingOut template parameter. Sometimes you may want to locally override the output encoding and you can do this by wrapping the object with WS::PutAs<desired encoding, wrapped type> (you can use WS::putAs function to construct WS::PutAs object without having to explicitly specify the second template parameter).

 WSStream<WS::Encoding::UTF8> mls { mlink };         // By default use UTF8
std::vector<std::string> vecOfExpr = ....;       // This is a vector of serialized Mathematica expressions,
ml << WS::putAs<WS::Encoding::Native>(vecOfExpr);    // it should be sent with Native encoding

Parameters

  • [in] wrp: - object to be sent

template<typename T>
WSStream &operator<<(const std::basic_string<T> &s) 🔗

Sends std::basic_string.

Template Parameters

  • T: - string character type supported in any of WSPut*String

Parameters

  • [in] s: - std::basic_string<T> to be sent

Exceptions

  • ErrorName::WSPutStringError:

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingStrings.html

template<typename T, std::size_t N, typename = std::enable_if_t<WS::StringTypeQ<T>>>
WSStream &operator<<(const T (&s)[N]) 🔗

Sends a character array (or a string literal)

Template Parameters

  • T: - character type supported in any of WSPut*String

  • N: - length of character array

Parameters

  • [in] s: - character array to be sent as String

Exceptions

  • ErrorName::WSPutStringError:

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingStrings.html

WSStream &operator<<(const char *s) 🔗

Sends a C-string.

Parameters

  • [in] s: - C-string to be sent

Exceptions

  • ErrorName::WSPutStringError:

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingStrings.html

template<typename K, typename V>
WSStream &operator<<(const std::map<K, V> &map) 🔗

Sends a std::map via WSTP, it is translated to an Association in Mathematica.

Template Parameters

  • K: - map key type, must be supported in WSStream

  • V: - map value type, must be supported in WSStream

Parameters

  • [in] map: - map to be sent as Association

Exceptions

  • ErrorName::WSPutFunctionError: plus whatever can be thrown sending keys and values

template<typename T, typename = std::enable_if_t<std::is_arithmetic_v<T>>>
WSStream &operator<<(T value) 🔗

Sends a scalar value (int, float, double, etc) if it is supported by WSTP If you need to send value of type not supported by WSTP (like unsigned int) you must either explicitly cast or provide your own overload.

Template Parameters

  • T: - scalar type

Parameters

  • [in] value: - numeric value to be sent

Exceptions

  • ErrorName::WSPutScalarError:

template<typename Container, typename = std::void_t<decltype(std::declval<Container>().begin(), std::declval<Container>().end(), std::declval<Container>().size())>>
WSStream &operator<<(const Container &c) 🔗

Sends any container (a class with begin(), end() and size()) as List.

Template Parameters

  • Container: - type that is a collection of some elements

Parameters

  • [in] c: - container to be sent

Exceptions

  • ErrorName::WSPutContainerError:

Note

Size() is not technically necessary, but needed for performance reason. Most STL containers have size() anyway.

WSStream &operator>>(BidirStreamToken f) 🔗

Receives a bidirectional stream token via WSTP.

Parameters

  • [in] f: - an element that can be either sent or received via WSTP with no arguments, for example WS::Rule

WSStream &operator>>(const WS::Symbol &s) 🔗

Receives a symbol from WSTP.

Parameter s must have head specified and it has to match the head that was read from WSTP

Parameters

  • [in] s: - a symbol

Exceptions

  • ErrorName::WSGetSymbolErrorErrorName::WSTestHeadError:

See

WS::Symbol

WSStream &operator>>(WS::Symbol &s) 🔗

Receives a symbol from WSTP.

If the parameter s has head specified, then it has to match the head that was read from WSTP, otherwise the head read from WSTP will be assigned to s

Parameters

  • [inout] s: - a symbol

Exceptions

  • ErrorName::WSGetSymbolErrorErrorName::WSTestHeadError:

See

WS::Symbol

WSStream &operator>>(const WS::Function &f) 🔗

Receives a function from WSTP.

Parameter f must have head and argument count specified and they need to match the head and argument count that was read from WSTP

Parameters

  • [in] f: - a function with head and argument count specified

Exceptions

  • ErrorName::WSGetFunctionErrorErrorName::WSTestHeadError:

See

WS::Function

WSStream &operator>>(WS::Function &f) 🔗

Receives a function from WSTP.

If the parameter f has head or argument count set, than it has to match the head or argument count that was read from WSTP

Parameters

  • [inout] f: - a function which may have head or argument count specified

Exceptions

  • ErrorName::WSGetFunctionErrorErrorName::WSTestHeadError:

See

WS::Function

WSStream &operator>>(bool &b) 🔗

Receives a True or False symbol from Mathematica and converts it to bool.

Parameters

  • [out] b: - argument to which the boolean received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetSymbolErrorErrorName::WSWrongSymbolForBool:

WSStream &operator>>(mint &i) 🔗

Receives a mint value.

Parameters

  • [in] i: - argument to which a mint value will be assigned

Note

It actually reads an wsint64 and casts to mint, as mint is not natively supported by WSTP.

template<typename T>
WSStream &operator>>(WS::ArrayData<T> &a) 🔗

Receives a WSTP array.

Template Parameters

  • T: - array element type

Parameters

  • [out] a: - argument to which the WS::ArrayData received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetArrayError:

See

WS::ArrayData<T>

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingMultidimensionalArrays.html

template<typename T>
WSStream &operator>>(WS::ListData<T> &l) 🔗

Receives a WSTP list.

Template Parameters

  • T: - list element type

Parameters

  • [out] l: - argument to which the WS::ListData received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetListError:

See

WS::ListData<T>

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingLists.html

template<typename T>
WSStream &operator>>(std::vector<T> &l) 🔗

Receives a List from WSTP and assigns it to std::vector.

Template Parameters

  • T: - vector element type (types supported in WSGet*List will be handled more efficiently)

Parameters

  • [out] l: - argument to which the List received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetListError:

template<WS::Encoding E = EncodingIn>
WSStream &operator>>(WS::StringData<E> &s) 🔗

Receives a WSTP string.

Template Parameters

  • T: - string character type

Parameters

  • [out] s: - argument to which the WS::StringData received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetStringError:

See

WS::StringData<T>

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingStrings.html

template<typename T>
WSStream &operator>>(std::basic_string<T> &s) 🔗

Receives std::basic_string.

Template Parameters

  • T: - string character type supported in any of WSGet*String

Parameters

  • [out] s: - argument to which the std::basic_string<T> received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetStringError:

See

http://reference.wolfram.com/language/guide/WSTPCFunctionsForExchangingStrings.html

Note

std::string is just std::basic_string<char>

template<WS::Encoding E, typename T>
WSStream &operator>>(WS::GetAs<E, T> wrp) 🔗

Receives a value of type T.

Template Parameters

  • E: - encoding to be used when reading value from WSTP

  • T: - value type

Parameters

  • wrp: - reference to object of type T wrapped in WS::GetAs structure

Note

There is a utility function WS::getAs for easier creation of WS::GetAs objects

template<typename K, typename V>
WSStream &operator>>(std::map<K, V> &map) 🔗

Receives a std::map via WSTP.

Template Parameters

  • K: - map key type, must be supported in WSStream

  • V: - map value type, must be supported in WSStream

Parameters

  • [out] map: - argument to which the std::map received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetFunctionError: plus whatever can be thrown receiving keys and values

Note

The top-level Association must have all values of the same type because this is how std::map works

template<typename T, typename = std::enable_if_t<std::is_arithmetic_v<T>>>
WSStream &operator>>(T &value) 🔗

Receives a scalar value (int, float, double, etc) if it is supported by WSTP If you need to receive value of type not supported by WSTP (like unsigned int) you must either explicitly cast or provide your own overload.

Template Parameters

  • T: - scalar type

Parameters

  • [out] value: - argument to which the value received from WSTP will be assigned

Exceptions

  • ErrorName::WSGetScalarError: