How to find all links pages on a website

Question

Is it possible to find all the pages and links on ANY given website  I d like to enter a URL and produce a directory tree of all links from that site   I ve looked at HTTrack but that downloads the whole site and I simply need the directory tree

User · Accepted Answer

Check out linkchecker   it will crawl the site  while obeying robots txt  and generate a report  From there  you can script up a solution for creating the directory tree

User · Answer

If you have the developer console  JavaScript  in your browser  you can type this code in   urls   document querySelectorAll  a    for  url in urls  console log urls url  href     Shortened   n     a   for u in n console log n u  href

User · Answer

function getalllinks  url         links   array        if   fp   fopen  url   r               content               while   line   fread  fp  1024                  content     line                       textLen   strlen  content       if   textLen  gt  10             startPos   0           valid   true          while   valid                 spos   strpos  content    lt a     startPos               if   spos  lt   startPos   valid   false               spos   strpos  content   href    spos                spos   strpos  content        spos    1               epos   strpos  content        spos                startPos    epos               link   substr  content   spos   epos -  spos               if  strpos  link   http          false   links      link                      return  links      try this code

User · Answer

Another alternative might be  Array from document querySelectorAll  a    map x   gt  x href    With your     its even shorter  Array from     a    map x   gt  x href

User · Answer

If this is a programming question  then I would suggest you write your own regular expression to parse all the retrieved contents  Target tags are IMG and A for standard HTML  For JAVA    final String openingTags      lt a    gt   href         lt img   gt     src             this along with Pattern and Matcher classes should detect the beginning of the tags  Add LINK tag if you also want CSS   However  it is not as easy as you may have intially thought  Many web pages are not well-formed  Extracting all the links programmatically that human being can  recognize  is really difficult if you need to take into account all the irregular expressions   Good luck

[directory] How to find all links / pages on a website

Examples related to directory

Examples related to web-crawler