WSTP support π
LibraryLink allows a LinkObject to be passed as an argument which may then exchange data between your library and the Kernel using
Wolfram Symbolic Transfer Protocol (WSTP, also known as MathLink).
The original WSTP is a C style API with error codes, macros, manual memory management, etc.
LLU provides a wrapper for the LinkObject called WSStream
.
WSStream
is actually a class template in the namespace LLU
parameterized by the default encodings to be used for strings, but for the sake of clarity,
both the template parameters and the namespace are skipped in the remainder of this text.
Main features π
Convenient syntax π
In LLU WSTP is interpreted as an I/O stream, so operators << and >> are utilized to make the syntax cleaner and more concise. This frees developers from the responsibility to choose the proper WSTP API function for the data they intend to read or write.
Error checking π
Each call to WSTP API has its return status checked. An exception is thrown on failures which carries some debug info to help locate the problem. Sample debug info looks like this:
Error code reported by WSTP: 48
"Unable to convert from given character encoding to WSTP encoding"
Additional debug info: WSPutUTF8String
Memory cleanup π
WSRelease* no longer needs to be called on the data received from WSTP. The LLU framework does it for you.
Automated handling of common data types π
Some sophisticated types can be sent to Wolfram Language directly via a WSStream class. For example nested maps:
std::map<std::string, std::map<int, std::vector<double>>> myNestedMap
Just write ms << myNestedMap and a nested Association will be returned. It works in the other direction too.
Obviously, for the above to work, the key and value types in the map must be supported by WSStream (i.e. there must exist an overload of
WSStream::operator<<
that takes an argument of given type).
User-defined classes π
Suppose you have a structure
struct Color {
double red;
double green;
double blue;
};
It is enough to overload operator<< like this:
1 2 3 | WSStream& operator<<(WSStream& ms, const Color& c) {
return ms << WS::Function("RGBColor", 3) << c.red << c.green << c.blue;
}
|
Objects of class Color can now be sent directly via WSStream.
Example π
Letβs compare the same piece of code written in plain LibraryLink with one written with LLU and WSStream. Here is the plain LibraryLink code:
if (!WSNewPacket(mlp)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutFunction(mlp, "List", nframes)) {
wsErr = -1;
goto cleanup;
}
for (auto& f : extractedFrames) {
if (!WSPutFunction(mlp, "List", 7)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutFunction(mlp, "Rule", 2)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutString(mlp, "ImageSize")) {
wsErr = -1;
goto cleanup;
}
if (!WSPutFunction(mlp, "List", 2)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutInteger64(mlp, f->width)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutInteger64(mlp, f->height)) {
wsErr = -1;
goto cleanup;
}
// ...
if (!WSPutFunction(mlp, "Rule", 2)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutString(mlp, "ImageOffset")) {
wsErr = -1;
goto cleanup;
}
if (!WSPutFunction(mlp, "List", 2)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutInteger64(mlp, f->left)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutInteger64(mlp, f->top)) {
wsErr = -1;
goto cleanup;
}
// ...
if (!WSPutFunction(mlp, "Rule", 2)) {
wsErr = -1;
goto cleanup;
}
if (!WSPutString(mlp, "UserInputFlag")) {
wsErr = -1;
goto cleanup;
}
if (!WSPutSymbol(mlp, f->userInputFlag == true ? "True" : "False")) {
wsErr = -1;
goto cleanup;
}
}
if (!WSEndPacket(mlp)) {
/* unable to send the end-of-packet sequence to mlp */
}
if (!WSFlush(mlp)){
/* unable to flush any buffered output data in mlp */
}
and now the same code using WSStream:
WSStream ms(mlp);
ms << WS::NewPacket;
ms << WS::List(nframes);
for (auto& f : extractedFrames) {
ms << WS::List(7)
<< WS::Rule
<< "ImageSize"
<< WS::List(2) << f->width << f->height
// ...
<< WS::Rule
<< "ImageOffset"
<< WS::List(2) << f->left << f->top
// ...
<< WS::Rule
<< "UserInputFlag"
<< f->userInputFlag
}
ms << WS::EndPacket << WS::Flush;
Expressions of unknown length π
Whenever you send an expression via WSTP you have to first specify the head and the number of arguments. This is not very flexible for example when an unknown number of contents are being read from a file.
As a workaround, one can create a temporary loopback link, accumulate all the arguments there (without the head), count the arguments, and then send everything to the βmainβ link as usual.
The same strategy has been incorporated into WSStream so that developers do not have to implement it. Now you can send a List like this:
1 2 3 4 5 6 7 | WSStream ms(mlp);
ms << WS::BeginExpr("List");
while (dataFromFile != EOF) {
// process data from file and send to WSStream
}
ms << WS::EndExpr();
|
Warning
This feature should only be used if necessary since it requires a temporary link and makes extra copies of data. Simple benchmarks showed a ~2x slowdown compared to the usual WSPutFunction.
API reference π
-
class
LLU
::
WSStream
π Wrapper class over WSTP with a stream-like interface.
WSStream resides in LLU namespace, whereas other WSTP-related classes can be found in LLU::WS namespace.
- Template Parameters
EncodingIn
: - default encoding to use when reading strings from WSTPEncodingOut
: - default encoding to use when writing strings to WSTP
Public Types
-
using
StreamToken
= WSStream &(*)(WSStream&) π Type of elements that can be sent via WSTP with no arguments, for example WS::Flush.
-
using
BidirStreamToken
= WSStream &(*)(WSStream&, WS::Direction) π Type of elements that can be either sent or received via WSTP with no arguments, for example WS::Rule.
-
using
LoopbackData
= std::pair<std::string, WSLINK> π Type of data stored on the stack to facilitate sending expressions of a priori unknown length.
Public Functions
-
WSStream
(WSLINK mlp, int argc) π Constructs new WSStream and checks whether there is a list of
argc
arguments on the LinkObject waiting to be read.
-
WSStream
(WSLINK mlp, const std::string &head, int argc) π Constructs new WSStream and checks whether there is a function with head
head
andargc
arguments on the LinkObject waiting to be read.
-
~WSStream
() = default π Default destructor.
-
WSLINK &
get
() noexcept π Returns a reference to underlying low-level WSTP handle.
-
template<typename
Iterator
, typename = enable_if_input_iterator<Iterator>>
voidsendRange
(Iterator begin, Iterator end) π Sends any range as List.
-
template<typename
Iterator
, typename = enable_if_input_iterator<Iterator>>
voidsendRange
(Iterator begin, Iterator end, const std::string &head) π Sends a range of elements as top-level expression with arbitrary head.
-
WSStream &
operator<<
(StreamToken f) π Sends a stream token via WSTP.
-
WSStream &
operator<<
(BidirStreamToken f) π Sends a bidirectional stream token via WSTP.
-
WSStream &
operator<<
(const WS::Function &f) π Sends a top-level function via WSTP, function arguments should be sent immediately after.
-
WSStream &
operator<<
(const WS::Missing &f) π Sends a top-level expression of the form Missing[βreasonβ].
-
WSStream &
operator<<
(const WS::BeginExpr &expr) π Starts sending a new expression where the number of arguments is not known a priori.
-
WSStream &
operator<<
(const WS::DropExpr &expr) π Drops current expression that was initiated with BeginExpr.
-
WSStream &
operator<<
(const WS::EndExpr &expr) π Ends current expression that was initiated with BeginExpr, prepends the head from BeginExpr and sends everything to the βparentβ link.
-
WSStream &
operator<<
(bool b) π Sends a boolean value via WSTP, it is translated to True or False in Mathematica.
-
template<typename
T
, typenameD
>
WSStream &operator<<
(const std::unique_ptr<T, D> &p) π Sends an object owned by unique pointer.
-
template<typename
T
>
WSStream &operator<<
(const std::vector<T> &l) π Sends a std::vector via WSTP, it is interpreted as a List in Mathematica.
-
template<WS::Encoding
E
, typenameT
>
WSStream &operator<<
(const WS::PutAs<E, T> &wrp) π Sends all strings within a given object using specified character encoding.
Normally, when you send a string WSStream chooses the appropriate WSTP function based on the EncodingOut template parameter. Sometimes you may want to locally override the output encoding and you can do this by wrapping the object with WS::PutAs<desired encoding, wrapped type> (you can use WS::putAs function to construct WS::PutAs object without having to explicitly specify the second template parameter).
WSStream<WS::Encoding::UTF8> mls { mlink }; // By default use UTF8 std::vector<std::string> vecOfExpr = ....; // This is a vector of serialized Mathematica expressions, ml << WS::putAs<WS::Encoding::Native>(vecOfExpr); // it should be sent with Native encoding
-
template<typename
T
>
WSStream &operator<<
(const std::basic_string<T> &s) π Sends std::basic_string.
-
template<typename
T
, std::size_tN
, typename = std::enable_if_t<WS::StringTypeQ<T>>>
WSStream &operator<<
(const T (&s)[N]) π Sends a character array (or a string literal)
-
template<typename
K
, typenameV
>
WSStream &operator<<
(const std::map<K, V> &map) π Sends a std::map via WSTP, it is translated to an Association in Mathematica.
-
template<typename
T
, typename = std::enable_if_t<std::is_arithmetic_v<T>>>
WSStream &operator<<
(T value) π Sends a scalar value (int, float, double, etc) if it is supported by WSTP If you need to send value of type not supported by WSTP (like unsigned int) you must either explicitly cast or provide your own overload.
-
template<typename
Container
, typename = std::void_t<decltype(std::declval<Container>().begin(), std::declval<Container>().end(), std::declval<Container>().size())>>
WSStream &operator<<
(const Container &c) π Sends any container (a class with begin(), end() and size()) as List.
-
WSStream &
operator>>
(BidirStreamToken f) π Receives a bidirectional stream token via WSTP.
-
WSStream &
operator>>
(const WS::Symbol &s) π Receives a symbol from WSTP.
Parameter
s
must have head specified and it has to match the head that was read from WSTP
-
WSStream &
operator>>
(WS::Symbol &s) π Receives a symbol from WSTP.
If the parameter
s
has head specified, then it has to match the head that was read from WSTP, otherwise the head read from WSTP will be assigned to s
-
WSStream &
operator>>
(const WS::Function &f) π Receives a function from WSTP.
Parameter
f
must have head and argument count specified and they need to match the head and argument count that was read from WSTP
-
WSStream &
operator>>
(WS::Function &f) π Receives a function from WSTP.
If the parameter
f
has head or argument count set, than it has to match the head or argument count that was read from WSTP
-
WSStream &
operator>>
(bool &b) π Receives a True or False symbol from Mathematica and converts it to bool.
-
template<typename
T
>
WSStream &operator>>
(std::vector<T> &l) π Receives a List from WSTP and assigns it to std::vector.
-
template<WS::Encoding
E
= EncodingIn>
WSStream &operator>>
(WS::StringData<E> &s) π Receives a WSTP string.
-
template<WS::Encoding
E
, typenameT
>
WSStream &operator>>
(WS::GetAs<E, T> wrp) π Receives a value of type T.
-
template<typename
K
, typenameV
>
WSStream &operator>>
(std::map<K, V> &map) π Receives a std::map via WSTP.
-
template<typename
T
, typename = std::enable_if_t<std::is_arithmetic_v<T>>>
WSStream &operator>>
(T &value) π Receives a scalar value (int, float, double, etc) if it is supported by WSTP If you need to receive value of type not supported by WSTP (like unsigned int) you must either explicitly cast or provide your own overload.