Clojure for fast processing of streams of data via LAZY-SEQ and SEQUE

UPDATE: After some back and forth I think clojure.core.async with its several buffers both at source, computation/transformation and sink areas is a better fit for what I'm doing! 

LAZY-SEQ and SEQUE are useful in many contexts where you need to speed up processing of streams of data (SEQ) from e.g. another computation, a filesystem or a web or database server -- anything really.

The key idea is that SEQUE will continuously fetch the next n few elements from the stream in advance and in the background (thread) -- while the code that consumes or handles the data keeps on working on the current or already fetched element(s) in the foreground.

A quick video to demonstrate the effect:

SEQUE uses a LinkedBlockingQueue behind the scenes, but you can pass it anything that implements the BlockingQueue interface as needed.

Clojure makes this simple and fun and all of this might be pretty basic and trivial for many, but a small "trick" is needed to set it up correctly -- like this:

Big data: from compressed text (e.g. CSV) to compressed binary format -- or why Nippy (Clojure) and java.io.DataOutputStream are awesome

Say you have massive amounts of historical market data in a common, gzip'ed CSV format or similar and you have these data types which represents instances of the data in your system:

(defrecord OFlow ;; Order flow; true trade and volume data!

    [^double trade ;; Positive = buy, negative = sell.

     ^double price ;; Average fill price.

     ^Keyword tick-direction ;; :plus | :zero-plus | :minus | :zero-minus

     ^long timestamp ;; We assume this is the ts for when the order executed in full.

     ^IMarketEvent memeta]

(defrecord MEMeta

    [^Keyword exchange-id

     ^String symbol

     ^long local-timestamp])

A good way to store and access this would be to use a binary format and a modern, fast compression algorithm. The key issue is fast decompression and LZ4HC is the best here as far as I'm aware of -- apparently reaching the limitations of what's possible with regards to RAM speed. To do this we'll use https://github.com/ptaoussanis/nippy which exposes the DataOutputStream class nicely and enables us to express a simple binary protocol for reading and writing our data types, like this:

(nippy/extend-freeze OFlow :QA/OFlow [^OFlow oflow output]

                     (.writeDouble output (.trade oflow))

                     (.writeDouble output (.price oflow))

                     (.writeByte output (case (.tick-direction oflow)

                                          :plus 0, :zero-plus 1, :minus 2, :zero-minus 3))

                     (.writeLong output (.timestamp oflow))

                     ;; MEMeta

                     (.writeUTF output (name (.exchange-id ^MEMeta (.memeta oflow))))

                     (.writeUTF output (.symbol ^MEMeta (.memeta oflow)))

                     (.writeLong output (.local-timestamp ^MEMeta (.memeta oflow))))

(nippy/extend-thaw :QA/OFlow [input]

                   (->OFlow (.readDouble input)

                            (.readDouble input)

                            (case (.readByte input)

                              0 :plus, 1 :zero-plus, 2 :minus, 3 :zero-minus)

                            (.readLong input)

                            (->MEMeta (keyword (.readUTF input))

                                      (.readUTF input)

                                      (.readLong input))))

..to write out the binary data to a file, you'd do something like (oflow-vector is a vector containing OFlow instances):

(nippy/freeze-to-file "data.dat" oflow-vector                                   

                      {:compressor nippy/lz4hc-compressor, :encryptor nil, :no-header? true})

..and to read it back in to get a vector of OFlow instances as the result you'd do something like this:

(nippy/thaw-from-file "data.dat"

                      {:compressor nippy/lz4hc-compressor, :encryptor nil, :no-header? true})

...it's so simple and the result is very, very good in terms of speed and space savings [I'll add some numbers here later]. Of course you'd still want to use something like PostgreSQL for indexed views or access to the data, but this is very nice for fast access to massive amounts of sequential, high resolution data. I've split things up in such a way that each file contains 1 day worth of data; this way it is possible to make fast requests to ranges of the data at any location without doing long, linear searches. 👍

A list of #mistakes update: you cannot process events in simple sequential order during live trading

NOTE: I'll keep adding more updates like this under the #mistakes label.

This might be pretty trivial and even obvious, but I keep having to remind myself of this as I "rediscover" it in several somewhat similar situations.

During backtesting you can pretend that you always get events one by one in perfect order and process them in sequence as such because you are in control of time here -- but during live trading you will sometimes get a bulk of events in one go and your strategy, order or event handling code will probably need to be aware of this in order to make good decisions.

E.g.:

1. You are passed a bulk of events.
2. An earlier event in the bulk informs you about an order update (e.g. partial fill) which you generate a response to.
3. A later event in the bulk informs you that the position related to that order was closed for whatever reason.
4. ...based on §3 your response in §2 might not make sense anymore.

You might think that this will not be a problem for you because your system will be fast enough to handle the events before they queue up anyway -- but this is not only about your system, but also about network and exchange/broker lag. I.e. you are not in control of time here!

As I learn more and more about "stream processing", the more multiple full passes of bulks of data seem to make sense. Sequential processing is an oversimplified view of reality --- it is still useful during simulations though.