Flow-based Programming

Panta rhei (Panta rhei) - Everything flows.

"Concat" Component

I am still trying to articulate the practical and theoretical differences between "classical" Flow-Based Programming (FBP), on the one hand, and NoFlo and the various FBP-inspired, synchronous, systems, on the other. An attempt can be found at FBP vs. FBP-inspired Systems.

The FBP paradigm and application development philosophy has been gathering steam over the last 50 years, and over the last few years a number of products have appeared, often described by their authors as being implementations of FBP, but usually lacking key characteristics of "classical" FBP, such as selective receive and "back pressure". JSFBP is an exception, but it does not appear to have much of a "future" (pun intended), and at best its threads are "green", rather than "red", so cannot support multiple cores. What is more serious, from my point of view, is that, if one of these products (apart from JSFBP, of course) is a person's first exposure to FBP, this may get in the way of their adapting to the all-important FBP "paradigm shift". And on the surface the differences can be quite subtle - it's what is going on underneath that matters. The "FBP-inspired" products do share some characteristics with "classical" FBP (component orientation, configurable modularity), but most of them miss the essence, which is that FBP is a paradigm change from the standard, traditional von Neumann paradigm to a much more "engineering style", component-oriented paradigm. This can be very confusing to people coming to FBP without adequate preparation (reading the book, building simple applications using one of the FBP implementations, etc.)

The FBP-like product that has gained a lot of attention to FBP was NoFlo, built using JavaScript and Node.js, but paradoxically NoFlo is one of the products that does not implement "classical" FBP, and therefore does not support the FBP Paradigm change, with all the productivity and maintainability gains that this provides. And more recently, Node-RED, which has basically the same problem.

"Classical" FBP (in what follows I will usually just say "FBP") actually has more in common with the old Unit Record technology, which was the way most data processing was done until about the late '60s, when the von Neumann machine started to take over. The von Neumann paradigm nowadays has programmers so brain-washed that they cannot conceive of a different way of doing data processing - yet FBP has many advantages over the von Neumann paradigm (the latter is still used within FBP components, so you don't have to give up on it all together!). When computers started to take over the field, we lost a number of subtle advantages of Unit Record, which weren't really appreciated at that time.

Bottom line: FBP applications are not von Neumann systems (although individual components are); they are more like factory assembly lines - i.e. a totally different paradigm!

I haven't really found a way to coax people to make the necessary paradigm shift - while at the same time sparing them the painful (?) task of reading at least the first 1/2 of my book ("Flow-Based Programming", 2nd ed.), so I thought I would try talking about a single component which seems to me to highlight a number of the differences.

I have chosen the FBP standard Concatenate component, as it is a simple general purpose FBP component with no parameters. Its function is to force two or more streams of data into a fixed sequence (one after another), usually for human consumption. An example of its use might be to produce a summary page from data coming from different (asynchronously running) areas of an FBP network - e.g. number of transactions might be generated by one part of the network, number of affected customers by another, and total run time by another... Let us then stipulate that they always have to be displayed in the same sequence, independent of when they were generated. I said "for human consumption" because humans usually like things to be displayed in a fixed sequence, even though machines don't care!


Concatenate is an example of a parameter-less FBP component. Its logic is essentially (in pseudocode):

     set variable "c" to array size for array input port
     initialize "i" to 0
     do while "i" < "c"
         receive all IPs from array input port element "i" until end of stream,
              emitting each one to the (single) output port
         increment "i"

With small amounts of data, the lack of back pressure in most FBP-inspired products probably doesn't hurt, but in an actual real-world situation you could easily accumulate a few billion IPs waiting at element 2, which cannot be processed until end of stream is detected at element 0 and at element 1. In "classical" FBP, you don't "receive" any of them until you are ready to process them. This works fine because, as well as selective receive, FBP has "back pressure", meaning that a sending process is suspended if the connection involved is full, so the processes generating the "element 2" data will just wait, without any special coding required on the part of the programmer.

Here is Concatenate in JSFBP:

  'use strict';

  module.exports = function concat() {
    var array = this.openInputPortArray('IN');
    var outport = this.openOutputPort('OUT');
    var ip = null;

    for (var i = 0; i < array.length; i++) {
      while (true) {
        ip = array[i].receive();
        if (ip === null) {

I could show this component in the other "classical" FBP languages (Java, C#, C++ (with Boost) or Lua), but they all have essentially the same structure, the big difference being that, under the covers, all of these implementations do support multiple cores (this doesn't affect the code, and is transparent to coders).

Now, I have copied the following from Ken Kan's GitHub site, written in CoffeeScript (bear in mind that CoffeeScript uses indentation to indicate logical nesting).

noflo = require 'noflo'

class Concat extends noflo.Component
  description: 'Gathers data from all incoming connections and sends
  them together in order of connection'
  constructor: ->
    @buffers = {}
    @hasConnected = {}

    @inPorts =
      in: new noflo.ArrayPort
    @outPorts =
      out: new noflo.Port

    subscribed = false
    @inPorts.in.on 'connect', (socket) =>
      @hasConnected[@inPorts.in.sockets.indexOf(socket)] = true

      # In this component we need to know which of the sockets
      # sent the data, so we connect to the sockets directly
      unless subscribed
        @subscribeSocket id for socket, id in @inPorts.in.sockets
        subscribed = true

    @inPorts.in.on 'begingroup', (group) =>
      @outPorts.out.beginGroup group
    @inPorts.in.on 'endgroup', =>
    @inPorts.in.on 'disconnect', =>
      # Check that all ports have disconnected before emitting
      for socket in @inPorts.in.sockets
        return if socket.isConnected()
      do @clearBuffers

  clearBuffers: ->
    for id, data of @buffers
      return unless @hasConnected[id]
    @buffers = {}
    @hasConnected = {}

  subscribeSocket: (id) ->
    @buffers[id] = []
    @inPorts.in.sockets[id].on 'data', (data) =>
      unless typeof @buffers[id] is 'object'
        @buffers[id] = []
      @buffers[id].push data
      do @checkSend

  checkSend: ->
    # First check that we have data in all buffers
    for socket, id in @inPorts.in.sockets
      # If any of the buffers is empty we cancel
      return unless @buffers[id]
      return unless @buffers[id].length

    # Okay, all buffers have data: send.
    @outPorts.out.send buffer.shift() for id, buffer of @buffers

exports.getComponent = -> new Concat

This does seem a lot more complicated, and it has a basic problem, not shared by classical FBP: all streams must be in storage before anything is output. This means that, unlike classical FBP, it cannot handle streams that won't fit into available storage. I know modern machines have massive amounts of storage, but philosophically it's very different! While this logic is probably fairly clear to someone "trained in the art", I find it totally opaque - maybe I am too close to "classical" FBP to easily understand it! However, when we compare the two, the CoffeeScript code shown above seems to support the conclusion that the attempt to support FBP in JavaScript (or CoffeeScript) will often lead to significantly increased code complexity.

Tom Robinson recently posted an implementation of Concat using async/await, which is included in ES2017 and supported natively in node.js and most browsers, as part of his Streampunk project. This does seem much closer to the spirit of classical FBP. As far as I know, however, Javascript still can't support multiple cores, which would seem to be a significant limitation if one wants to use JavaScript on the server side...

FBP vs. FBP-inspired Systems