Combining node js and Python

Question

Node js is a perfect match for our web project  but there are few computational tasks for which we would prefer Python  We also already have a Python code for them  We are highly concerned about speed  what is the most elegant way how to call a Python  worker  from node js in an asynchronous non-blocking way

User · Answer

I ve had a lot of success using thoonk js along with thoonk py  Thoonk leverages Redis  in-memory key-value store  to give you feed  think publish subscribe   queue and job patterns for communication   Why is this better than unix sockets or direct tcp sockets  Overall performance may be decreased a little  however Thoonk provides a really simple API that simplifies having to manually deal with a socket  Thoonk also helps make it really trivial to implement a distributed computing model that allows you to scale your python workers to increase performance  since you just spin up new instances of your python workers and connect them to the same redis server

User · Answer

I d recommend using some work queue using  for example  the excellent Gearman  which will provide you with a great way to dispatch background jobs  and asynchronously get their result once they re processed   The advantage of this  used heavily at Digg  among many others  is that it provides a strong  scalable and robust way to make workers in any language to speak with clients in any language

User · Answer

I d consider also Apache Thrift http   thrift apache org   It can bridge between several programming languages  is highly efficient and has support for async or sync calls  See full features here http   thrift apache org docs features   The multi language can be useful for future plans  for example if you later want to do part of the computational task in C   it s very easy to do add it to the mix using Thrift

User · Answer

For communication between node js and Python server  I would use Unix sockets if both processes run on the same server and TCP IP sockets otherwise  For marshaling protocol I would take JSON or protocol buffer  If threaded Python shows up to be a bottleneck  consider using Twisted Python  which provides the same event driven concurrency as do node js   If you feel adventurous  learn clojure  clojurescript  clojure-py  and you ll get the same language that runs and interoperates with existing code on Java  JavaScript  node js included   CLR and Python  And you get superb marshalling protocol by simply using clojure data structures

User · Answer

Update 2019  There are several ways to achieve this and here is the list in increasing order of complexity   Python Shell  you will write streams to the python console and it will write back to you  Redis Pub Sub  you can have a channel     listening in Python while your node js publisher pushes data  Websocket connection where Node acts as the client and Python acts        as the server or vice-versa  API connection with Express Flask Tornado etc working separately with an API endpoint exposed for the other to query   Approach 1 Python Shell Simplest approach   source js file  const ps   require  python-shell      very important to add -u option since our python script runs infinitely var options         pythonPath    Users zup  local share virtualenvs python shell test-TJN5lQez bin python       pythonOptions    -u       get print results in real-time        make sure you use an absolute path for scriptPath     scriptPath     subscriber           args    value1    value2    value3        mode   json      const shell   new ps PythonShell  destination py   options    function generateArray         const list          for  let i   0  i  lt  1000  i              list push Math random     1000            return list    setInterval      gt        shell send generateArray       1000    shell on  message   message   gt        console log message        destination py file  import datetime import sys import time import numpy import talib import timeit import json import logging logging basicConfig format    asctime s     levelname s     message s   level logging INFO   size   1000 p   100 o   numpy random random size  h   numpy random random size  l   numpy random random size  c   numpy random random size  v   numpy random random size   def get indicators values         Return the RSI of the values sent from node js     numpy values   numpy array values  dtype numpy double       return talib func RSI numpy values  14   for line in sys stdin      l   json loads line      print get indicators l         Without this step the output may not be immediately available in node     sys stdout flush     Notes  Make a folder called subscriber which is at the same level as source js file and put destination py inside it  Dont forget to change your virtualenv environment

User · Answer

If you arrange to have your Python worker in a separate process  either long-running server-type process or a spawned child on demand   your communication with it will be asynchronous on the node js side  UNIX TCP sockets and stdin out err communication are inherently async in node

User · Answer

This sounds like a scenario where zeroMQ would be a good fit  It s a messaging framework that s similar to using TCP or Unix sockets  but it s much more robust  http   zguide zeromq org py all   There s a library that uses zeroMQ to provide a RPC framework that works pretty well  It s called zeroRPC  http   www zerorpc io    Here s the hello world    Python  Hello x  server   import zerorpc  class HelloRPC object          pass the method a name  it replies  Hello name          def hello self  name           return  Hello   0    format name   def main        s   zerorpc Server HelloRPC        s bind  tcp     4242       s run    if   name         main      main     And the node js client   var zerorpc   require  zerorpc     var client   new zerorpc Client    client connect  tcp   127 0 0 1 4242      calls the method on the python object client invoke  hello    World   function error  reply  streaming        if error           console log  ERROR     error             console log reply         Or vice-versa  node js server   var zerorpc   require  zerorpc     var server   new zerorpc Server       hello  function name  reply            reply null   Hello      name  false              server bind  tcp   0 0 0 0 4242      And the python client  import zerorpc  sys  c   zerorpc Client   c connect  tcp   127 0 0 1 4242   name   sys argv 1  if len sys argv   gt  1 else  dude  print c hello name

[python] Combining node.js and Python

Examples related to python

Examples related to node.js

Examples related to ipc