Go-like concurrency (and profiling!) in Hy and PyPy

Waking up at 6:30 AM has bestowed upon me a blissful, irritant-free period that is perfect for contemplation, so of course I used it for a few experiments instead.

With all the hoopla around Go concurrency, a surprisingly large number of folk have been pursuing similar avenues for writing nice, clean concurrent code, and analogues have sprung up all over the place – like Clojure‘s core.async, for instance, which is around a year old and which I’m quite fond of.

In Python land, one of the available options is Rob Galanakis’ goless, which I’ve been tracking fairly closely. It provides Go-like primitives and works wonderfully out-of-the-box with PyPy, providing in-process coroutines with pretty much zero hassle.

As it happens, PyPy is now my default Python interpreter¹, so goless is a very nice addition to my toolkit, all the more so now that I’m starting to benchmark and profile most of my code as a matter of course – partly to figure out where the bottlenecks are, and partly to figure out ways to make it go faster².

And it’s been quite entertaining, really – I’ve found a few interesting things here and there, and profile results from even trivial bits of code is quite informative.

For instance, here’s one of the goless examples (the very first one I picked up, really), rewritten in Hy (and with inline profiling until #618 is addressed):

; A straight port of the goless library example (runs fine with the PyPy backend)
(import [goless [go chan]]
        [cProfile [Profile]]
        [pstats [Stats]]
        [functools [partial]])

(defn produce [msgs done count]
    (for [i (xrange count)]
        (.send msgs i))
    (.send done))

(defn consume [msgs out name]
    (for [msg msgs]
        (.send out (% "%s:%s" (, name msg)))))

(defn logger [out]
    (for [msg out]
        (print msg)))

(let [[p (Profile)]
      [done (chan)]
      [msgs (chan)]
      [out  (chan)]]
      ; enable profiler
      (.enable p)
      ; start a producer, three consumers and a logger
      (go produce msgs done 10000)
      (map (partial go consume msgs out) ["one" "two" "three"])
      (go logger out)
      ; wait for completion, stop profiler and dump stats
      (.recv done)
      ; stop profiler
      (.disable p)
      (.dump_stats (Stats p) "out.pstats"))

…and here’s the profile result (in SVG format), courtesy of gprof2dot:

Next up, applying this to a number of things I’m building around an MQTT broker…