WSTP support πŸ”—

LibraryLink allows a LinkObject to be passed as an argument which may then exchange data between your library and the Kernel using Wolfram Symbolic Transfer Protocol (WSTP, also known as MathLink). The original WSTP is a C style API with error codes, macros, manual memory management, etc. LLU provides a wrapper for the LinkObject called WSStream.

WSStream is actually a class template in the namespace LLU parameterized by the default encodings to be used for strings, but for the sake of clarity, both the template parameters and the namespace are skipped in the remainder of this text.

Main features πŸ”—

Convenient syntax πŸ”—

In LLU WSTP is interpreted as an I/O stream, so operators << and >> are utilized to make the syntax cleaner and more concise. This frees developers from the responsibility to choose the proper WSTP API function for the data they intend to read or write.

Error checking πŸ”—

Each call to WSTP API has its return status checked. An exception is thrown on failures which carries some debug info to help locate the problem. Sample debug info looks like this:

Error code reported by WSTP: 48
"Unable to convert from given character encoding to WSTP encoding"
Additional debug info: WSPutUTF8String

Memory cleanup πŸ”—

WSRelease* no longer needs to be called on the data received from WSTP. The LLU framework does it for you.

Automated handling of common data types πŸ”—

Some sophisticated types can be sent to Wolfram Language directly via a WSStream class. For example nested maps:

std::map<std::string, std::map<int, std::vector<double>>> myNestedMap

Just write ms << myNestedMap and a nested Association will be returned. It works in the other direction too. Obviously, for the above to work, the key and value types in the map must be supported by WSStream (i.e. there must exist an overload of WSStream::operator<< that takes an argument of given type).

User-defined classes πŸ”—

Suppose you have a structure

struct Color {
    double red;
    double green;
    double blue;
};

It is enough to overload operator<< like this:

1
2
3
    WSStream& operator<<(WSStream& ms, const Color& c) {
        return ms << WS::Function("RGBColor", 3) << c.red << c.green << c.blue;
    }

Objects of class Color can now be sent directly via WSStream.

Example πŸ”—

Let’s compare the same piece of code written in plain LibraryLink with one written with LLU and WSStream. Here is the plain LibraryLink code:

    if (!WSNewPacket(mlp)) {
        wsErr = -1;
        goto cleanup;
    }
    if (!WSPutFunction(mlp, "List", nframes)) {
        wsErr = -1;
        goto cleanup;
    }
    for (auto& f : extractedFrames) {
        if (!WSPutFunction(mlp, "List", 7)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutFunction(mlp, "Rule", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutString(mlp, "ImageSize")) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutFunction(mlp, "List", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->width)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->height)) {
            wsErr = -1;
            goto cleanup;
        }
        // ...
        if (!WSPutFunction(mlp, "Rule", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutString(mlp, "ImageOffset")) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutFunction(mlp, "List", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->left)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutInteger64(mlp, f->top)) {
            wsErr = -1;
            goto cleanup;
        }
        // ...
        if (!WSPutFunction(mlp, "Rule", 2)) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutString(mlp, "UserInputFlag")) {
            wsErr = -1;
            goto cleanup;
        }
        if (!WSPutSymbol(mlp, f->userInputFlag == true ? "True" : "False")) {
            wsErr = -1;
            goto cleanup;
        }
    }
    if (!WSEndPacket(mlp)) {
            /* unable to send the end-of-packet sequence to mlp */
    }
    if (!WSFlush(mlp)){
            /* unable to flush any buffered output data in mlp */
    }

and now the same code using WSStream:

    WSStream ms(mlp);

    ms << WS::NewPacket;
    ms << WS::List(nframes);

    for (auto& f : extractedFrames) {
        ms << WS::List(7)
            << WS::Rule
                << "ImageSize"
                << WS::List(2) << f->width << f->height
            // ...
            << WS::Rule
                << "ImageOffset"
                << WS::List(2) << f->left << f->top
            // ...
            << WS::Rule
                << "UserInputFlag"
                << f->userInputFlag
    }

    ms << WS::EndPacket << WS::Flush;

Expressions of unknown length πŸ”—

Whenever you send an expression via WSTP you have to first specify the head and the number of arguments. This is not very flexible for example when an unknown number of contents are being read from a file.

As a workaround, one can create a temporary loopback link, accumulate all the arguments there (without the head), count the arguments, and then send everything to the β€œmain” link as usual.

The same strategy has been incorporated into WSStream so that developers do not have to implement it. Now you can send a List like this:

1
2
3
4
5
6
7
    WSStream ms(mlp);

    ms << WS::BeginExpr("List");
    while (dataFromFile != EOF) {
            // process data from file and send to WSStream
    }
    ms << WS::EndExpr();

Warning

This feature should only be used if necessary since it requires a temporary link and makes extra copies of data. Simple benchmarks showed a ~2x slowdown compared to the usual WSPutFunction.

API reference πŸ”—

class LLU::WSStream πŸ”—

Wrapper class over WSTP with a stream-like interface.

WSStream resides in LLU namespace, whereas other WSTP-related classes can be found in LLU::WS namespace.

Template Parameters
  • EncodingIn: - default encoding to use when reading strings from WSTP

  • EncodingOut: - default encoding to use when writing strings to WSTP

Public Types

using StreamToken = WSStream &(*)(WSStream&) πŸ”—

Type of elements that can be sent via WSTP with no arguments, for example WS::Flush.

using BidirStreamToken = WSStream &(*)(WSStream&, WS::Direction) πŸ”—

Type of elements that can be either sent or received via WSTP with no arguments, for example WS::Rule.

using LoopbackData = std::pair<std::string, WSLINK> πŸ”—

Type of data stored on the stack to facilitate sending expressions of a priori unknown length.

Public Functions

Constructs new WSStream.

WSStream(WSLINK mlp, int argc) πŸ”—β–Ό

Constructs new WSStream and checks whether there is a list of argc arguments on the LinkObject waiting to be read.

WSStream(WSLINK mlp, const std::string &head, int argc) πŸ”—β–Ό

Constructs new WSStream and checks whether there is a function with head head and argc arguments on the LinkObject waiting to be read.

~WSStream() = default πŸ”—

Default destructor.

WSLINK &get() noexcept πŸ”—

Returns a reference to underlying low-level WSTP handle.

template<typename Iterator, typename = enable_if_input_iterator<Iterator>>
void sendRange(Iterator begin, Iterator end) πŸ”—β–Ό

Sends any range as List.

template<typename Iterator, typename = enable_if_input_iterator<Iterator>>
void sendRange(Iterator begin, Iterator end, const std::string &head) πŸ”—β–Ό

Sends a range of elements as top-level expression with arbitrary head.

WSStream &operator<<(StreamToken f) πŸ”—β–Ό

Sends a stream token via WSTP.

WSStream &operator<<(BidirStreamToken f) πŸ”—β–Ό

Sends a bidirectional stream token via WSTP.

WSStream &operator<<(const WS::Symbol &s) πŸ”—β–Ό

Sends a top-level symbol via WSTP.

WSStream &operator<<(const WS::Function &f) πŸ”—β–Ό

Sends a top-level function via WSTP, function arguments should be sent immediately after.

WSStream &operator<<(const WS::Missing &f) πŸ”—β–Ό

Sends a top-level expression of the form Missing[β€œreason”].

WSStream &operator<<(const WS::BeginExpr &expr) πŸ”—β–Ό

Starts sending a new expression where the number of arguments is not known a priori.

WSStream &operator<<(const WS::DropExpr &expr) πŸ”—β–Ό

Drops current expression that was initiated with BeginExpr.

WSStream &operator<<(const WS::EndExpr &expr) πŸ”—β–Ό

Ends current expression that was initiated with BeginExpr, prepends the head from BeginExpr and sends everything to the β€œparent” link.

WSStream &operator<<(bool b) πŸ”—β–Ό

Sends a boolean value via WSTP, it is translated to True or False in Mathematica.

WSStream &operator<<(mint i) πŸ”—β–Ό

Sends a mint value.

template<typename T>
WSStream &operator<<(const WS::ArrayData<T> &a) πŸ”—β–Ό

Sends a WSTP array.

template<typename T>
WSStream &operator<<(const WS::ListData<T> &l) πŸ”—β–Ό

Sends a WSTP list.

template<typename T, typename D>
WSStream &operator<<(const std::unique_ptr<T, D> &p) πŸ”—β–Ό

Sends an object owned by unique pointer.

template<typename T>
WSStream &operator<<(const std::vector<T> &l) πŸ”—β–Ό

Sends a std::vector via WSTP, it is interpreted as a List in Mathematica.

template<WS::Encoding E>
WSStream &operator<<(const WS::StringData<E> &s) πŸ”—β–Ό

Sends a WSTP string.

template<WS::Encoding E, typename T>
WSStream &operator<<(const WS::PutAs<E, T> &wrp) πŸ”—β–Ό

Sends all strings within a given object using specified character encoding.

Normally, when you send a string WSStream chooses the appropriate WSTP function based on the EncodingOut template parameter. Sometimes you may want to locally override the output encoding and you can do this by wrapping the object with WS::PutAs<desired encoding, wrapped type> (you can use WS::putAs function to construct WS::PutAs object without having to explicitly specify the second template parameter).

 WSStream<WS::Encoding::UTF8> mls { mlink };         // By default use UTF8
std::vector<std::string> vecOfExpr = ....;       // This is a vector of serialized Mathematica expressions,
ml << WS::putAs<WS::Encoding::Native>(vecOfExpr);    // it should be sent with Native encoding

template<typename T>
WSStream &operator<<(const std::basic_string<T> &s) πŸ”—β–Ό

Sends std::basic_string.

template<typename T, std::size_t N, typename = std::enable_if_t<WS::StringTypeQ<T>>>
WSStream &operator<<(const T (&s)[N]) πŸ”—β–Ό

Sends a character array (or a string literal)

WSStream &operator<<(const char *s) πŸ”—β–Ό

Sends a C-string.

template<typename K, typename V>
WSStream &operator<<(const std::map<K, V> &map) πŸ”—β–Ό

Sends a std::map via WSTP, it is translated to an Association in Mathematica.

template<typename T, typename = std::enable_if_t<std::is_arithmetic_v<T>>>
WSStream &operator<<(T value) πŸ”—β–Ό

Sends a scalar value (int, float, double, etc) if it is supported by WSTP If you need to send value of type not supported by WSTP (like unsigned int) you must either explicitly cast or provide your own overload.

template<typename Container, typename = std::void_t<decltype(std::declval<Container>().begin(), std::declval<Container>().end(), std::declval<Container>().size())>>
WSStream &operator<<(const Container &c) πŸ”—β–Ό

Sends any container (a class with begin(), end() and size()) as List.

WSStream &operator>>(BidirStreamToken f) πŸ”—β–Ό

Receives a bidirectional stream token via WSTP.

WSStream &operator>>(const WS::Symbol &s) πŸ”—β–Ό

Receives a symbol from WSTP.

Parameter s must have head specified and it has to match the head that was read from WSTP

WSStream &operator>>(WS::Symbol &s) πŸ”—β–Ό

Receives a symbol from WSTP.

If the parameter s has head specified, then it has to match the head that was read from WSTP, otherwise the head read from WSTP will be assigned to s

WSStream &operator>>(const WS::Function &f) πŸ”—β–Ό

Receives a function from WSTP.

Parameter f must have head and argument count specified and they need to match the head and argument count that was read from WSTP

WSStream &operator>>(WS::Function &f) πŸ”—β–Ό

Receives a function from WSTP.

If the parameter f has head or argument count set, than it has to match the head or argument count that was read from WSTP

WSStream &operator>>(bool &b) πŸ”—β–Ό

Receives a True or False symbol from Mathematica and converts it to bool.

WSStream &operator>>(mint &i) πŸ”—β–Ό

Receives a mint value.

template<typename T>
WSStream &operator>>(WS::ArrayData<T> &a) πŸ”—β–Ό

Receives a WSTP array.

template<typename T>
WSStream &operator>>(WS::ListData<T> &l) πŸ”—β–Ό

Receives a WSTP list.

template<typename T>
WSStream &operator>>(std::vector<T> &l) πŸ”—β–Ό

Receives a List from WSTP and assigns it to std::vector.

template<WS::Encoding E = EncodingIn>
WSStream &operator>>(WS::StringData<E> &s) πŸ”—β–Ό

Receives a WSTP string.

template<typename T>
WSStream &operator>>(std::basic_string<T> &s) πŸ”—β–Ό

Receives std::basic_string.

template<WS::Encoding E, typename T>
WSStream &operator>>(WS::GetAs<E, T> wrp) πŸ”—β–Ό

Receives a value of type T.

template<typename K, typename V>
WSStream &operator>>(std::map<K, V> &map) πŸ”—β–Ό

Receives a std::map via WSTP.

template<typename T, typename = std::enable_if_t<std::is_arithmetic_v<T>>>
WSStream &operator>>(T &value) πŸ”—β–Ό

Receives a scalar value (int, float, double, etc) if it is supported by WSTP If you need to receive value of type not supported by WSTP (like unsigned int) you must either explicitly cast or provide your own overload.