Executing Javascript from Python

Question

I have HTML webpages that I am crawling using xpath  The etree tostring of a certain node gives me this string    lt script gt   lt  -- function escramble 758      var a b c   a   1     b  84-    a   425-    b   7450    c  9    document write a c b    escramble 758     -- gt   lt  script gt    I just need the output of escramble 758    I can write a regex to figure out the whole thing  but I want my code to remain tidy  What is the best alternative   I am zipping through the following libraries  but I didnt see an exact solution  Most of them are trying to emulate browser  making things snail slow    http   code google com p python-spidermonkey   clearly says it s not yet possible to call a function defined in Javascript  http   code google com p webscraping   don t see anything for Javascript  I may be wrong  http   pypi python org pypi selenium  Emulating browser    Edit  An example will be great    barebones will do

User · Answer

You can use js2py context to execute your js code and get output from document write with mock document object   import js2py  js       var output  document         write  function value           output   value                your script  context   js2py EvalJs   context execute js  print context output

User · Answer

Using PyV8  I can do this  However  I have to replace document write with return because there s no DOM and therefore no document   import PyV8 ctx   PyV8 JSContext   ctx enter    js       function escramble 758    var a b c a   1   b  84-  a   425-  b   7450  c  9  document write a c b    escramble 758        print ctx eval js replace  document write    return       Or you could create a mock document object  class MockDocument object        def   init   self           self value           def write self   args           self value       join str i  for i in args    class Global PyV8 JSClass       def   init   self           self document   MockDocument    scope   Global   ctx   PyV8 JSContext scope  ctx enter   ctx eval js  print scope document value

User · Answer

You can use requests-html which will download and use chromium underneath   from requests html import HTML  html   HTML html   lt a href  http   www example com   gt     script       function escramble 758        var a b c     a   1       b  84-      a   425-      b   7450      c  9      return a c b         val   html render script script  reload False  print val     1 425-984-7450   More on this read here

User · Answer

You can also use Js2Py which is written in pure python and is able to both execute and translate javascript to python  Supports virtually whole JavaScript even labels  getters  setters and other rarely used features    import js2py  js       function escramble 758    var a b c a   1   b  84-  a   425-  b   7450  c  9  document write a c b    escramble 758       replace  document write    return     result   js2py eval js js     executing JavaScript and converting the result to python string    Advantages of Js2Py include portability and extremely easy integration with python  since basically JavaScript is being translated to python     To install   pip install js2py

User · Answer

quickjs should be the best option after quickjs come out   Just pip install quickjs and you are ready to go   modify based on the example on README   from quickjs import Function  js       function escramble 758    var a b c a   1   b  84-  a   425-  b   7450  c  9  document write a c b  escramble 758          escramble 758   Function  escramble 758   js replace  document write    return      print escramble 758      https   github com PetterS quickjs

User · Answer

One more solution as PyV8 seems to be unmaintained and dependent on the old version of libv8    PyMiniRacer It s a wrapper around the v8 engine and it works with the new version and is actively maintained   pip install py-mini-racer  from py mini racer import py mini racer ctx   py mini racer MiniRacer   ctx eval     function escramble 758        var a b c     a   1       b  84-      a   425-      b   7450      c  9      return a c b         ctx call  escramble 758     And yes  you have to replace document write with return as others suggested

[javascript] Executing Javascript from Python

Examples related to javascript

Examples related to python

Examples related to screen-scraping