Hadoop cluster setup - java net ConnectException Connection refused

Question

I want to setup a hadoop-cluster in pseudo-distributed mode  I managed to perform all the setup-steps  including startuping a Namenode  Datanode  Jobtracker and a Tasktracker on my machine   Then I tried to run some exemplary programms and faced the java net ConnectException  Connection refused error  I stepped back to the very first steps of running some operations in standalone mode and faced the same problem    I performed even triple-check of all the installation steps and have no idea how to fix it   I am new to Hadoop and a beginner Ubuntu user thus I kindly ask you for  taking it into account  if providing any guide or tip      This is the error output I keep receiving    hduser marta-komputer  usr local hadoop  bin hadoop jar share hadoop mapreduce hadoop-mapreduce-examples-2 6 0 jar grep input output  dfs a-z     15 02 22 18 23 04 WARN util NativeCodeLoader  Unable to load native-hadoop library for your platform    using builtin-java classes where applicable 15 02 22 18 23 04 INFO client RMProxy  Connecting to ResourceManager at  0 0 0 0 8032 java net ConnectException  Call From marta-komputer 127 0 1 1 to localhost 9000 failed on connection exception  java net ConnectException  Connection refused  For more details see   http   wiki apache org hadoop ConnectionRefused     at sun reflect NativeConstructorAccessorImpl newInstance0 Native Method      at sun reflect NativeConstructorAccessorImpl newInstance NativeConstructorAccessorImpl java 62      at sun reflect DelegatingConstructorAccessorImpl newInstance DelegatingConstructorAccessorImpl java 45      at java lang reflect Constructor newInstance Constructor java 408      at org apache hadoop net NetUtils wrapWithMessage NetUtils java 791      at org apache hadoop net NetUtils wrapException NetUtils java 731      at org apache hadoop ipc Client call Client java 1472      at org apache hadoop ipc Client call Client java 1399      at org apache hadoop ipc ProtobufRpcEngine Invoker invoke ProtobufRpcEngine java 232      at com sun proxy  Proxy9 delete Unknown Source      at org apache hadoop hdfs protocolPB ClientNamenodeProtocolTranslatorPB delete ClientNamenodeProtocolTranslatorPB java 521      at sun reflect NativeMethodAccessorImpl invoke0 Native Method      at sun reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62      at sun reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43      at java lang reflect Method invoke Method java 483      at org apache hadoop io retry RetryInvocationHandler invokeMethod RetryInvocationHandler java 187      at org apache hadoop io retry RetryInvocationHandler invoke RetryInvocationHandler java 102      at com sun proxy  Proxy10 delete Unknown Source      at org apache hadoop hdfs DFSClient delete DFSClient java 1929      at org apache hadoop hdfs DistributedFileSystem 12 doCall DistributedFileSystem java 638      at org apache hadoop hdfs DistributedFileSystem 12 doCall DistributedFileSystem java 634      at org apache hadoop fs FileSystemLinkResolver resolve FileSystemLinkResolver java 81      at org apache hadoop hdfs DistributedFileSystem delete DistributedFileSystem java 634      at org apache hadoop examples Grep run Grep java 95      at org apache hadoop util ToolRunner run ToolRunner java 70      at org apache hadoop examples Grep main Grep java 101      at sun reflect NativeMethodAccessorImpl invoke0 Native Method      at sun reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62      at sun reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43      at java lang reflect Method invoke Method java 483      at org apache hadoop util ProgramDriver ProgramDescription invoke ProgramDriver java 71      at org apache hadoop util ProgramDriver run ProgramDriver java 144      at org apache hadoop examples ExampleDriver main ExampleDriver java 74      at sun reflect NativeMethodAccessorImpl invoke0 Native Method      at sun reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62      at sun reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43      at java lang reflect Method invoke Method java 483      at org apache hadoop util RunJar run RunJar java 221      at org apache hadoop util RunJar main RunJar java 136  Caused by  java net ConnectException  Connection refused     at sun nio ch SocketChannelImpl checkConnect Native Method      at sun nio ch SocketChannelImpl finishConnect SocketChannelImpl java 716      at org apache hadoop net SocketIOWithTimeout connect SocketIOWithTimeout java 206      at org apache hadoop net NetUtils connect NetUtils java 530      at org apache hadoop net NetUtils connect NetUtils java 494      at org apache hadoop ipc Client Connection setupConnection Client java 607      at org apache hadoop ipc Client Connection setupIOstreams Client java 705      at org apache hadoop ipc Client Connection access 2800 Client java 368      at org apache hadoop ipc Client getConnection Client java 1521      at org apache hadoop ipc Client call Client java 1438          32 more     etc hadoop hadoop-env sh file     The java implementation to use  export JAVA HOME  usr lib jvm java-8-oracle    The jsvc implementation to use  Jsvc is required to run secure datanodes   that bind to privileged ports to provide authentication of data transfer   protocol   Jsvc is not required if SASL is configured for authentication of   data transfer protocol using non-privileged ports   export JSVC HOME   JSVC HOME   export HADOOP CONF DIR   HADOOP CONF DIR -  etc hadoop      Extra Java CLASSPATH elements   Automatically insert capacity-scheduler  for f in  HADOOP HOME contrib capacity-scheduler   jar  do   if     HADOOP CLASSPATH     then     export HADOOP CLASSPATH  HADOOP CLASSPATH  f   else     export HADOOP CLASSPATH  f   fi done    The maximum amount of heap to use  in MB  Default is 1000   export HADOOP HEAPSIZE   export HADOOP NAMENODE INIT HEAPSIZE       Extra Java runtime options   Empty by default  export HADOOP OPTS   HADOOP OPTS -Djava net preferIPv4Stack true     Command specific options appended to HADOOP OPTS when specified export HADOOP NAMENODE OPTS  -Dhadoop security logger   HADOOP SECURITY LOGGER -INFO RFAS  -Dhdfs audit logger   HDFS AUDIT LOGGER -INFO NullAppender   HADOOP NAMENODE OPTS  export HADOOP DATANODE OPTS  -Dhadoop security logger ERROR RFAS  HADOOP DATANODE OPTS   export HADOOP SECONDARYNAMENODE OPTS  -Dhadoop security logger   HADOOP SECURITY LOGGER -INFO RFAS  -Dhdfs audit logger   HDFS AUDIT LOGGER -INFO NullAppender   HADOOP SECONDARYNAMENODE OPTS   export HADOOP NFS3 OPTS   HADOOP NFS3 OPTS  export HADOOP PORTMAP OPTS  -Xmx512m  HADOOP PORTMAP OPTS     The following applies to multiple commands  fs  dfs  fsck  distcp etc  export HADOOP CLIENT OPTS  -Xmx512m  HADOOP CLIENT OPTS   HADOOP JAVA PLATFORM OPTS  -XX -UsePerfData  HADOOP JAVA PLATFORM OPTS     On secure datanodes  user to run the datanode as after dropping privileges    This   MUST   be uncommented to enable secure HDFS if using privileged ports   to provide authentication of data transfer protocol   This   MUST NOT   be   defined if SASL is configured for authentication of data transfer protocol   using non-privileged ports  export HADOOP SECURE DN USER   HADOOP SECURE DN USER     Where log files are stored    HADOOP HOME logs by default   export HADOOP LOG DIR   HADOOP LOG DIR   USER    Where log files are stored in the secure data environment  export HADOOP SECURE DN LOG DIR   HADOOP LOG DIR    HADOOP HDFS USER     HDFS Mover specific parameters       Specify the JVM options to be used when starting the HDFS Mover    These options will be appended to the options specified as HADOOP OPTS   and therefore may override any similar flags set in HADOOP OPTS     export HADOOP MOVER OPTS           Advanced Users Only         The directory where pid files are stored   tmp by default    NOTE  this should be set to a directory that can only be written to by          the user that will run the hadoop daemons   Otherwise there is the         potential for a symlink attack  export HADOOP PID DIR   HADOOP PID DIR  export HADOOP SECURE DN PID DIR   HADOOP PID DIR     A string representing this instance of hadoop   USER by default  export HADOOP IDENT STRING  USER    bashrc file Hadoop-related fragment     -- HADOOP ENVIRONMENT VARIABLES START --   export JAVA HOME  usr lib jvm java-8-oracle export HADOOP HOME  usr local hadoop export PATH  PATH  HADOOP HOME bin export PATH  PATH  HADOOP HOME sbin export HADOOP MAPRED HOME  HADOOP HOME export HADOOP COMMON HOME  HADOOP HOME export HADOOP HDFS HOME  HADOOP HOME export YARN HOME  HADOOP HOME export HADOOP COMMON LIB NATIVE DIR  HADOOP HOME lib native export HADOOP OPTS  -Djava library path  HADOOP HOME lib    -- HADOOP ENVIRONMENT VARIABLES END --      usr local hadoop etc hadoop core-site xml file    lt configuration gt    lt property gt     lt name gt hadoop tmp dir lt  name gt     lt value gt  usr local hadoop tmp lt  value gt     lt description gt A base for other temporary directories  lt  description gt   lt  property gt    lt property gt   lt name gt fs default name lt  name gt   lt value gt hdfs   localhost 9000 lt  value gt   lt  property gt    lt  configuration gt     usr local hadoop etc hadoop hdfs-site xml file    lt configuration gt   lt property gt         lt name gt dfs replication lt  name gt         lt value gt 1 lt  value gt    lt  property gt    lt property gt         lt name gt dfs namenode name dir lt  name gt         lt value gt file  usr local hadoop tmp hdfs namenode lt  value gt    lt  property gt    lt property gt         lt name gt dfs datanode data dir lt  name gt         lt value gt file  usr local hadoop tmp hdfs datanode lt  value gt    lt  property gt   lt  configuration gt     usr local hadoop etc hadoop yarn-site xml file    lt configuration gt    lt property gt         lt name gt yarn nodemanager aux-services lt  name gt         lt value gt mapreduce shuffle lt  value gt   lt  property gt   lt property gt         lt name gt yarn nodemanager aux-services mapreduce shuffle class lt  name gt         lt value gt org apache hadoop mapred ShuffleHandler lt  value gt   lt  property gt   lt  configuration gt     usr local hadoop etc hadoop mapred-site xml file    lt configuration gt   lt property gt         lt name gt mapreduce framework name lt  name gt         lt value gt yarn lt  value gt   lt  property gt   lt configuration gt    Running hduser marta-komputer  usr local hadoop  bin hdfs namenode -format results in an output as follows  I substitiute some of its part with           hduser marta-komputer  usr local hadoop  bin hdfs namenode -format 15 02 22 18 50 47 INFO namenode NameNode  STARTUP MSG                                                                 STARTUP MSG  Starting NameNode STARTUP MSG    host   marta-komputer 127 0 1 1 STARTUP MSG    args    -format  STARTUP MSG    version   2 6 0 STARTUP MSG    classpath    usr local hadoop etc hadoop  usr local hadoop share hadoop common lib htrace-core-3 0 4 jar  usr local hadoop share hadoop common lib commons-cli      2 6 0 jar  usr local hadoop share hadoop mapreduce hadoop-mapreduce-client-common-2 6 0 jar  usr local hadoop share hadoop mapreduce hadoop-mapreduce-client-hs-2 6 0 jar  usr local hadoop share hadoop mapreduce hadoop-mapreduce-client-app-2 6 0 jar  usr local hadoop share hadoop mapreduce hadoop-mapreduce-client-shuffle-2 6 0 jar  usr local hadoop contrib capacity-scheduler   jar STARTUP MSG    build   https   git-wip-us apache org repos asf hadoop git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1  compiled by  jenkins  on 2014-11-13T21 10Z STARTUP MSG    java   1 8 0 31                                                               15 02 22 18 50 47 INFO namenode NameNode  registered UNIX signal handlers for  TERM  HUP  INT  15 02 22 18 50 47 INFO namenode NameNode  createNameNode  -format  15 02 22 18 50 47 WARN util NativeCodeLoader  Unable to load native-hadoop library for your platform    using builtin-java classes where applicable Formatting using clusterid  CID-0b65621a-eab3-47a4-bfd0-62b5596a940c 15 02 22 18 50 48 INFO namenode FSNamesystem  No KeyProvider found  15 02 22 18 50 48 INFO namenode FSNamesystem  fsLock is fair true 15 02 22 18 50 48 INFO blockmanagement DatanodeManager  dfs block invalidate limit 1000 15 02 22 18 50 48 INFO blockmanagement DatanodeManager  dfs namenode datanode registration ip-hostname-check true 15 02 22 18 50 48 INFO blockmanagement BlockManager  dfs namenode startup delay block deletion sec is set to 000 00 00 00 000 15 02 22 18 50 48 INFO blockmanagement BlockManager  The block deletion will start around 2015 Feb 22 18 50 48 15 02 22 18 50 48 INFO util GSet  Computing capacity for map BlocksMap 15 02 22 18 50 48 INFO util GSet  VM type         64-bit 15 02 22 18 50 48 INFO util GSet  2 0  max memory 889 MB   17 8 MB 15 02 22 18 50 48 INFO util GSet  capacity        2 21   2097152 entries 15 02 22 18 50 48 INFO blockmanagement BlockManager  dfs block access token enable false 15 02 22 18 50 48 INFO blockmanagement BlockManager  defaultReplication           1 15 02 22 18 50 48 INFO blockmanagement BlockManager  maxReplication               512 15 02 22 18 50 48 INFO blockmanagement BlockManager  minReplication               1 15 02 22 18 50 48 INFO blockmanagement BlockManager  maxReplicationStreams        2 15 02 22 18 50 48 INFO blockmanagement BlockManager  shouldCheckForEnoughRacks    false 15 02 22 18 50 48 INFO blockmanagement BlockManager  replicationRecheckInterval   3000 15 02 22 18 50 48 INFO blockmanagement BlockManager  encryptDataTransfer          false 15 02 22 18 50 48 INFO blockmanagement BlockManager  maxNumBlocksToLog            1000 15 02 22 18 50 48 INFO namenode FSNamesystem  fsOwner               hduser  auth SIMPLE  15 02 22 18 50 48 INFO namenode FSNamesystem  supergroup            supergroup 15 02 22 18 50 48 INFO namenode FSNamesystem  isPermissionEnabled   true 15 02 22 18 50 48 INFO namenode FSNamesystem  HA Enabled  false 15 02 22 18 50 48 INFO namenode FSNamesystem  Append Enabled  true 15 02 22 18 50 48 INFO util GSet  Computing capacity for map INodeMap 15 02 22 18 50 48 INFO util GSet  VM type         64-bit 15 02 22 18 50 48 INFO util GSet  1 0  max memory 889 MB   8 9 MB 15 02 22 18 50 48 INFO util GSet  capacity        2 20   1048576 entries 15 02 22 18 50 48 INFO namenode NameNode  Caching file names occuring more than 10 times 15 02 22 18 50 48 INFO util GSet  Computing capacity for map cachedBlocks 15 02 22 18 50 48 INFO util GSet  VM type         64-bit 15 02 22 18 50 48 INFO util GSet  0 25  max memory 889 MB   2 2 MB 15 02 22 18 50 48 INFO util GSet  capacity        2 18   262144 entries 15 02 22 18 50 48 INFO namenode FSNamesystem  dfs namenode safemode threshold-pct   0 9990000128746033 15 02 22 18 50 48 INFO namenode FSNamesystem  dfs namenode safemode min datanodes   0 15 02 22 18 50 48 INFO namenode FSNamesystem  dfs namenode safemode extension       30000 15 02 22 18 50 48 INFO namenode FSNamesystem  Retry cache on namenode is enabled 15 02 22 18 50 48 INFO namenode FSNamesystem  Retry cache will use 0 03 of total heap and retry cache entry expiry time is 600000 millis 15 02 22 18 50 48 INFO util GSet  Computing capacity for map NameNodeRetryCache 15 02 22 18 50 48 INFO util GSet  VM type         64-bit 15 02 22 18 50 48 INFO util GSet  0 029999999329447746  max memory 889 MB   273 1 KB 15 02 22 18 50 48 INFO util GSet  capacity        2 15   32768 entries 15 02 22 18 50 48 INFO namenode NNConf  ACLs enabled  false 15 02 22 18 50 48 INFO namenode NNConf  XAttrs enabled  true 15 02 22 18 50 48 INFO namenode NNConf  Maximum size of an xattr  16384 Re-format filesystem in Storage Directory  usr local hadoop tmp hdfs namenode    Y or N  Y 15 02 22 18 50 50 INFO namenode FSImage  Allocated new BlockPoolId  BP-948369552-127 0 1 1-1424627450316 15 02 22 18 50 50 INFO common Storage  Storage directory  usr local hadoop tmp hdfs namenode has been successfully formatted  15 02 22 18 50 50 INFO namenode NNStorageRetentionManager  Going to retain 1 images with txid  gt   0 15 02 22 18 50 50 INFO util ExitUtil  Exiting with status 0 15 02 22 18 50 50 INFO namenode NameNode  SHUTDOWN MSG                                                                 SHUTDOWN MSG  Shutting down NameNode at marta-komputer 127 0 1 1                                                                 Starting dfs and yarn results in the following output   hduser marta-komputer  usr local hadoop  start-dfs sh 15 02 22 18 53 05 WARN util NativeCodeLoader  Unable to load native-hadoop library for your platform    using builtin-java classes where applicable Starting namenodes on  localhost  localhost  starting namenode  logging to  usr local hadoop logs hadoop-hduser-namenode-marta-komputer out localhost  starting datanode  logging to  usr local hadoop logs hadoop-hduser-datanode-marta-komputer out Starting secondary namenodes  0 0 0 0  0 0 0 0  starting secondarynamenode  logging to  usr local hadoop logs hadoop-hduser-secondarynamenode-marta-komputer out 15 02 22 18 53 20 WARN util NativeCodeLoader  Unable to load native-hadoop library for your platform    using builtin-java classes where applicable hduser marta-komputer  usr local hadoop  start-yarn sh starting yarn daemons starting resourcemanager  logging to  usr local hadoop logs yarn-hduser-resourcemanager-marta-komputer out localhost  starting nodemanager  logging to  usr local hadoop logs yarn-hduser-nodemanager-marta-komputer out   Calling jps shortly after that gives   hduser marta-komputer  usr local hadoop  jps 11696 ResourceManager 11842 NodeManager 11171 NameNode 11523 SecondaryNameNode 12167 Jps   netstat output   hduser marta-komputer  usr local hadoop  sudo netstat -lpten   grep java tcp        0      0 0 0 0 0 8088            0 0 0 0                 LISTEN      1001       690283      11696 java       tcp        0      0 0 0 0 0 42745           0 0 0 0                 LISTEN      1001       684574      11842 java       tcp        0      0 0 0 0 0 13562           0 0 0 0                 LISTEN      1001       680955      11842 java       tcp        0      0 0 0 0 0 8030            0 0 0 0                 LISTEN      1001       684531      11696 java       tcp        0      0 0 0 0 0 8031            0 0 0 0                 LISTEN      1001       684524      11696 java       tcp        0      0 0 0 0 0 8032            0 0 0 0                 LISTEN      1001       680879      11696 java       tcp        0      0 0 0 0 0 8033            0 0 0 0                 LISTEN      1001       687392      11696 java       tcp        0      0 0 0 0 0 8040            0 0 0 0                 LISTEN      1001       680951      11842 java       tcp        0      0 127 0 0 1 9000          0 0 0 0                 LISTEN      1001       687242      11171 java       tcp        0      0 0 0 0 0 8042            0 0 0 0                 LISTEN      1001       680956      11842 java       tcp        0      0 0 0 0 0 50090           0 0 0 0                 LISTEN      1001       690252      11523 java       tcp        0      0 0 0 0 0 50070           0 0 0 0                 LISTEN      1001       687239      11171 java      etc hosts file   127 0 0 1       localhost 127 0 1 1       marta-komputer    The following lines are desirable for IPv6 capable hosts   1     ip6-localhost ip6-loopback fe00  0 ip6-localnet ff00  0 ip6-mcastprefix ff02  1 ip6-allnodes ff02  2 ip6-allrouters                                                         UPDATE 1   I updated the core-site xml and now I have    lt property gt   lt name gt fs default name lt  name gt   lt value gt hdfs   marta-komputer 9000 lt  value gt   lt  property gt    but I keep receiving the error - now starting as   15 03 01 00 59 34 INFO client RMProxy  Connecting to ResourceManager at  0 0 0 0 8032 java net ConnectException  Call From marta-komputer home 192 168 1 8 to marta-komputer 9000 failed on connection exception      java net ConnectException  Connection refused  For more details see     http   wiki apache org hadoop ConnectionRefused   I also notice that telnet localhost 9000 is not working   hduser marta-komputer    telnet localhost 9000 Trying 127 0 0 1    telnet  Unable to connect to remote host  Connection refused

User · Answer

From the netstat output you can see the process is listening on address 127 0 0 1  tcp        0      0 127 0 0 1 9000          0 0 0 0          from the exception message you can see that it tries to connect to address 127 0 1 1  java net ConnectException  Call From marta-komputer 127 0 1 1 to localhost 9000 failed       further in the exception it s mentionend  For more details see   http   wiki apache org hadoop ConnectionRefused   on this page you find     Check that there isn t an entry for your hostname mapped to 127 0 0 1 or 127 0 1 1 in  etc hosts  Ubuntu is notorious for this    so the conclusion is to remove this line in your  etc hosts  127 0 1 1       marta-komputer

User · Answer

Check your firewall setting  and  set      lt property gt     lt name gt fs default name lt  name gt     lt value gt hdfs   MachineName 9000 lt  value gt     lt  property gt    replace localhost to machine name

User · Answer

hduser marta-komputer  usr local hadoop  jps      11696 ResourceManager      11842 NodeManager      11171 NameNode      11523 SecondaryNameNode      12167 Jps   Where is your DataNode  Connection refused problem might also be due to no active DataNode  Check datanode logs for issues   UPDATED   For this error      15 03 01 00 59 34 INFO client RMProxy  Connecting to ResourceManager at  0 0 0 0 8032   java net ConnectException  Call From marta-komputer home 192 168 1 8 to marta-komputer 9000 failed on connection exception      java net ConnectException  Connection refused  For more details see     http   wiki apache org hadoop ConnectionRefused   Add these lines in yarn-site xml    lt property gt   lt name gt yarn resourcemanager address lt  name gt   lt value gt 192 168 1 8 8032 lt  value gt   lt  property gt   lt property gt   lt name gt yarn resourcemanager scheduler address lt  name gt   lt value gt 192 168 1 8 8030 lt  value gt   lt  property gt   lt property gt   lt name gt yarn resourcemanager resource-tracker address lt  name gt   lt value gt 192 168 1 8 8031 lt  value gt   lt  property gt    Restart the hadoop processes

User · Answer

Stop it by-  stop-all sh  format the namenode-  hadoop namenode -format again start-  start-all sh

User · Answer

For me these steps worked   stop-all sh hadoop namenode -format start-all sh

User · Answer

I resolved the same issue by adding this property to hdfs-site xml   lt property gt     lt name gt fs default name lt  name gt     lt value gt hdfs   localhost 9000 lt  value gt     lt  property gt

User · Answer

Make sure HDFS is online  Start it by  HADOOP HOME sbin start-dfs sh Once you do that  your test with telnet localhost 9001should work

User · Answer

In  etc hosts    Add this line    your-ip-address    your-host-name  example  192 168 1 8    master  In  etc hosts    Delete the line with 127 0 1 1  This will cause loopback  In your core-site  change localhost to your-ip or your-hostname   Now  restart the cluster

User · Answer

get in  SPARK HOME conf  then open file spark-env sh and add  SPARK MASTER HOST  your-IP  SPARK LOCAL IP 127 0 0 1

User · Answer

I am also facing same issue in Hortonworks   At the time I restart the Ambari agents and servers then the issue has been resolved    systemctl stop ambari-agent   systemctl stop ambari-server   Source  Full Article With Resolution   systemctl start ambari-agent  systemctl start ambari-server

User · Answer

In my experaince  15 02 22 18 23 04 WARN util NativeCodeLoader  Unable to load native-hadoop library for your platform    using builtin-java classes where applicable   You may have 64 bit version OS  and hadoop installation 32bit  refer this  java net ConnectException  Call From marta-komputer 127 0 1 1 to localhost 9000 failed on connection exception  java net ConnectException   connection refused  For more details see     http   wiki apache org hadoop ConnectionRefused   this problem refers to your ssh public key authorization  please provide details about your ssh set up   Please refer this link to check the complete steps    also provide info if  cat  HOME  ssh authorized keys   returns any result or not

User · Answer

For me it was that I could not cluster my zookeeper   hdfs haadmin -getServiceState 1 active  hdfs haadmin -getServiceState 2 active   My hadoop-hdfs-zkfc- hostname  log showed      2017-04-14 11 46 55 351 WARN org apache hadoop ha HealthMonitor    Transport-level exception trying to monitor health of NameNode at   HOST 192 168 1 55 9000  java net ConnectException  Connection refused   Call From HOST 192 168 1 55 to HOST 9000 failed on connection   exception  java net ConnectException  Connection refused  For more   details see   http   wiki apache org hadoop ConnectionRefused   solution   hdfs-site xml    lt property gt       lt name gt dfs namenode rpc-bind-host lt  name gt         lt value gt 0 0 0 0 lt  value gt     lt  property gt    before  netstat -plunt  tcp        0      0 192 168 1 55 9000        0 0 0 0                 LISTEN      13133 java  nmap localhost -p 9000  Starting Nmap 6 40   http   nmap org   at 2017-04-14 12 15 EDT Nmap scan report for localhost  127 0 0 1  Host is up  0 000047s latency   Other addresses for localhost  not scanned   127 0 0 1 PORT     STATE  SERVICE 9000 tcp closed cslistener   after  netstat -plunt tcp        0      0 0 0 0 0 9000            0 0 0 0                 LISTEN      14372 java  nmap localhost -p 9000  Starting Nmap 6 40   http   nmap org   at 2017-04-14 12 28 EDT Nmap scan report for localhost  127 0 0 1  Host is up  0 000039s latency   Other addresses for localhost  not scanned   127 0 0 1 PORT     STATE SERVICE 9000 tcp open  cslistener

User · Answer

Hi Edit your conf core-site xml and change localhost to 0 0 0 0  Use the conf below  That should work    lt configuration gt     lt property gt    lt name gt fs default name lt  name gt    lt value gt hdfs   0 0 0 0 9000 lt  value gt   lt  property gt

User · Answer

Your issue is a very interesting one  Hadoop setup could be frustrating some time due to the complexity of the system and many moving parts involved  I think the issue you faced is definitely a firewall one  My hadoop cluster has similar setup  With a firewall rule added with command    sudo iptables -A INPUT -p tcp --dport 9000 -j REJECT   I m able to see the exact issue   15 03 02 23 46 10 INFO client RMProxy  Connecting to ResourceManager at   0 0 0 0 8032 java net ConnectException  Call From mybox 127 0 1 1 to localhost 9000 failed on connection exception  java net ConnectException  Connection refused  For more details see   http   wiki apache org hadoop ConnectionRefused      at sun reflect NativeConstructorAccessorImpl newInstance0 Native Method    You can verify your firewall settings with command    usr local hadoop etc  sudo iptables -L Chain INPUT  policy ACCEPT  target     prot opt source               destination          REJECT     tcp  --  anywhere             anywhere             tcp dpt 9000 reject-with icmp-port-unreachable  Chain FORWARD  policy ACCEPT  target     prot opt source               destination           Chain OUTPUT  policy ACCEPT  target     prot opt source               destination      Once the suspicious rule is identified  it could be deleted with a command like    sudo iptables -D INPUT -p tcp --dport 9000 -j REJECT    Now  the connection should go through

User · Answer

I had the similar prolem with OP  As the terminal output suggested  I went to  http   wiki apache org hadoop ConnectionRefused    I tried to change my  etc hosts file as suggested here  i e  remove 127 0 1 1 as OP suggested it will create another error    So in the end  I leave it as is  The following is my  etc hosts   127 0 0 1       localhost localdomain   localhost 127 0 1 1       linux   The following lines are desirable for IPv6 capable hosts   1     ip6-localhost ip6-loopback fe00  0 ip6-localnet ff00  0 ip6-mcastprefix ff02  1 ip6-allnodes ff02  2 ip6-allrouters   In the end  I found that my namenode did not started correctly  i e  When you type sudo netstat -lpten   grep java in the terminal  there will not be any JVM process running listening  on port 9000   So I made two directories for namenode and datanode respectively if you have not done so   You don t have to put where I put it  please replace it based on your hadoop directory   i e   mkdir -p  home hadoopuser hadoop-2 6 2 hdfs namenode mkdir -p  home hadoopuser hadoop-2 6 2 hdfs datanode   I reconfigured my hdfs-site xml    lt configuration gt       lt property gt           lt name gt dfs replication lt  name gt           lt value gt 1 lt  value gt       lt  property gt      lt property gt           lt name gt dfs namenode name dir lt  name gt           lt value gt file  home hadoopuser hadoop-2 6 2 hdfs namenode lt  value gt       lt  property gt       lt property gt           lt name gt dfs datanode data dir lt  name gt           lt value gt file  home hadoopuser hadoop-2 6 2 hdfs datanode lt  value gt       lt  property gt   lt  configuration gt    In terminal  stop your hdfs and yarn with script stop-dfs sh and stop-yarn sh  They are located in your hadoop directory sbin  In my case  it s  home hadoopuser hadoop-2 6 2 sbin    Then start your hdfs and yarn with script start-dfs sh and start-yarn sh After it is started  type jps in your terminal to see if your JVM processes are running correctly  It should show the following   15678 NodeManager 14982 NameNode 15347 SecondaryNameNode 23814 Jps 15119 DataNode 15548 ResourceManager   Then try to use netstat again to see if your namenode is listening to port 9000  sudo netstat -lpten   grep java   If you successfully set up the namenode  you should see the following in your terminal output   tcp        0      0 127 0 0 1 9000          0 0 0 0                 LISTEN      1001       175157      14982 java  Then try to type the command hdfs dfs -mkdir  user hadoopuser If this command executes sucessfully  now you can list your directory in the HDFS user directory by hdfs dfs -ls  user

User · Answer

I was getting the same issue and found that OpenSSH service was not running and it was causing the issue  After starting the SSH service it worked  To check if SSH service is running or not  ssh localhost  To start the service  if OpenSSH is already installed  sudo  etc init d ssh start

[java] Hadoop cluster setup - java.net.ConnectException: Connection refused

Examples related to java

Examples related to hadoop

Examples related to configuration

Examples related to connectexception