How do search engines deal with AngularJS applications

Question

I see two issues with AngularJS application regarding search engines and SEO   1  What happens with custom tags  Do search engines ignore the whole content within those tags  i e  suppose I have   lt custom gt     lt h1 gt Hey  this title is important lt  h1 gt   lt  custom gt    would  lt h1 gt  be indexed despite being inside custom tags     2  Is there a way to avoid search engines of indexing      binds literally  i e    lt h2 gt   title   lt  h2 gt    I know I could do something like   lt h2 ng-bind  title  gt  lt  h2 gt    but what if I want to actually let the crawler  see  the title  Is server-side rendering the only solution

User · Answer

Crawlers  or bots  are designed to crawl HTML content of web pages but due to AJAX operations for asynchronous data fetching  this became a problem as it takes sometime to render page and show dynamic content on it  Similarly  AngularJS also use asynchronous model  which creates problem for Google crawlers   Some developers create basic html pages with real data and serve these pages from server side at the time of crawling  We can render same pages with PhantomJS on serve side which has  escaped fragment   Because Google looks for    in our site urls and then takes everything after the    and adds it in  escaped fragment  query parameter   For more detail please read this blog

User · Answer

Use PushState and Precomposition  The current  2015  way to do this is using the JavaScript pushState method   PushState changes the URL in the top browser bar without reloading the page  Say you have a page containing tabs  The tabs hide and show content  and the content is inserted dynamically  either using AJAX or by simply setting display none and display block to hide and show the correct tab content   When the tabs are clicked  use pushState to update the url in the address bar  When the page is rendered  use the value in the address bar to determine which tab to show  Angular routing will do this for you automatically   Precomposition  There are two ways to hit a PushState Single Page App  SPA    Via PushState  where the user clicks a PushState link and the content is AJAXed in  By hitting the URL directly    The initial hit on the site will involve hitting the URL directly  Subsequent hits will simply AJAX in content as the PushState updates the URL   Crawlers harvest links from a page then add them to a queue for later processing  This means that for a crawler  every hit on the server is a direct hit  they don t navigate via Pushstate   Precomposition bundles the initial payload into the first response from the server  possibly as a JSON object  This allows the Search Engine to render the page without executing the AJAX call   There is some evidence to suggest that Google might not execute AJAX requests  More on this here   https   web archive org web 20160318211223 http   www analog-ni co precomposing-a-spa-may-become-the-holy-grail-to-seo  Search Engines can read and execute JavaScript  Google has been able to parse JavaScript for some time now  it s why they originally developed Chrome  to act as a full featured headless browser for the Google spider  If a link has a valid href attribute  the new URL can be indexed  There s nothing more to do   If clicking a link in addition triggers a pushState call  the site can be navigated by the user via PushState   Search Engine Support for PushState URLs  PushState is currently supported by Google and Bing   Google  Here s Matt Cutts responding to Paul Irish s question about PushState for SEO   http   youtu be yiAF9VdvRPw  Here is Google announcing full JavaScript support for the spider   http   googlewebmastercentral blogspot de 2014 05 understanding-web-pages-better html  The upshot is that Google supports PushState and will index PushState URLs   See also Google webmaster tools  fetch as Googlebot  You will see your JavaScript  including Angular  is executed    Bing  Here is Bing s announcement of support for pretty PushState URLs dated March 2013   http   blogs bing com webmaster 2013 03 21 search-engine-optimization-best-practices-for-ajax-urls   Don t use HashBangs     Hashbang urls were an ugly stopgap requiring the developer to provide a pre-rendered version of the site at a special location  They still work  but you don t need to use them   Hashbang URLs look like this   domain com   path to resource  This would be paired with a metatag like this    lt meta name  fragment  content     gt   Google will not index them in this form  but will instead pull a static version of the site from the  escaped fragments  URL and index that    Pushstate URLs look like any ordinary URL   domain com path to resource  The difference is that Angular handles them for you by intercepting the change to document location dealing with it in JavaScript   If you want to use PushState URLs  and you probably do  take out all the old hash style URLs and metatags and simply enable HTML5 mode in your config block   Testing your site  Google Webmaster tools now contains a tool which will allow you to fetch a URL as google  and render JavaScript as Google renders it   https   www google com webmasters tools googlebot-fetch  Generating PushState URLs in Angular  To generate real URLs in Angular  rather than   prefixed ones  set HTML5 mode on your  locationProvider object    locationProvider html5Mode true     Server Side  Since you are using real URLs  you will need to ensure the same template  plus some precomposed content  gets shipped by your server for all valid URLs  How you do this will vary depending on your server architecture   Sitemap  Your app may use unusual forms of navigation  for example hover or scroll  To ensure Google is able to drive your app  I would probably suggest creating a sitemap  a simple list of all the urls your app responds to  You can place this at the default location   sitemap or  sitemap xml   or tell Google about it using webmaster tools   It s a good idea to have a sitemap anyway   Browser support  Pushstate works in IE10  In older browsers  Angular will automatically fall back to hash style URLs  A demo page  The following content is rendered using a pushstate URL with precomposition   http   html5 gingerhost com london  As can be verified  at this link  the content is indexed and is appearing in Google    Serving 404 and 301 Header status codes  Because the search engine will always hit your server for every request  you can serve header status codes from your server and expect Google to see them

User · Answer

Update May 2014  Google crawlers now executes javascript - you can use the Google Webmaster Tools to better understand how your sites are rendered by Google     Original answer If you want to optimize your app for search engines there is unfortunately no way around serving a pre-rendered version to the crawler  You can read more about Google s recommendations for ajax and javascript-heavy sites here   If this is an option I d recommend reading this article about how to do SEO for Angular with server-side rendering    I   m not sure what the crawler does when it encounters custom tags

User · Answer

Google s Crawlable Ajax Spec  as referenced in the other answers here  is basically the answer   If you re interested in how other search engines and social bots deal with the same issues I wrote up the state of art here  http   blog ajaxsnapshots com 2013 11 googles-crawlable-ajax-specification html  I work for a https   ajaxsnapshots com  a company that implements the Crawlable Ajax Spec as a service - the information in that report is based on observations from our logs

User · Answer

As of now Google has changed their AJAX crawling proposal      Times have changed  Today  as long as you re not blocking Googlebot from crawling your JavaScript or CSS files  we are generally able to render and understand your web pages like modern browsers    tl dr   Google  are no longer recommending the AJAX crawling proposal  Google  made back in 2009

User · Answer

This has drastically changed   http   searchengineland com bing-offers-recommendations-for-seo-friendly-ajax-suggests-html5-pushstate-152946  If you use   locationProvider html5Mode true   you are set    No more rendering pages

User · Answer

You should really check out the tutorial on building an SEO-friendly AngularJS site on the year of moo blog  He walks you through all the steps outlined on Angular s documentation  http   www yearofmoo com 2012 11 angularjs-and-seo html  Using this technique  the search engine sees the expanded HTML instead of the custom tags

User · Answer

Let s get definitive about AngularJS and SEO  Google  Yahoo  Bing  and other search engines crawl the web in traditional ways using traditional crawlers  They run robots that crawl the HTML on web pages  collecting information along the way  They keep interesting words and look for other links to other pages  these links  the amount of them and the number of them come into play with SEO     So why don t search engines deal with javascript sites   The answer has to do with the fact that the search engine robots work through headless browsers and they most often do not have a javascript rendering engine to render the javascript of a page  This works for most pages as most static pages don t care about JavaScript rendering their page  as their content is already available   What can be done about it   Luckily  crawlers of the larger sites have started to implement a mechanism that allows us to make our JavaScript sites crawlable  but it requires us to implement a change to our site   If we change our hashPrefix to be    instead of simply    then modern search engines will change the request to use  escaped fragment  instead of      With HTML5 mode  i e  where we have links without the hash prefix  we can implement this same feature by looking at the User Agent header in our backend    That is to say  instead of a request from a normal browser that looks like   http   www ng-newsletter com    signup page  A search engine will search the page with   http   www ng-newsletter com   escaped fragment   signup page  We can set the hash prefix of our Angular apps using a built-in method from ngRoute   angular module  myApp        config    location   function  location       location hashPrefix              And  if we re using html5Mode  we will need to implement this using the meta tag    lt meta name  fragment  content     gt    Reminder  we can set the html5Mode   with the  location service   angular module  myApp        config    location    function  location       location html5Mode true          Handling the search engine  We have a lot of opportunities to determine how we ll deal with actually delivering content to search engines as static HTML  We can host a backend ourselves  we can use a service to host a back-end for us  we can use a proxy to deliver the content  etc  Let s look at a few options   Self-hosted  We can write a service to handle dealing with crawling our own site using a headless browser  like phantomjs or zombiejs  taking a snapshot of the page with rendered data and storing it as HTML  Whenever we see the query string   escaped fragment  in a search request  we can deliver the static HTML snapshot we took of the page instead of the pre-rendered page through only JS  This requires us to have a backend that delivers our pages with conditional logic in the middle  We can use something like prerender io s backend as a starting point to run this ourselves  Of course  we still need to handle the proxying and the snippet handling  but it s a good start   With a paid service  The easiest and the fastest way to get content into search engine is to use a service Brombone  seo js  seo4ajax  and prerender io are good examples of these that will host the above content rendering for you  This is a good option for the times when we don t want to deal with running a server proxy  Also  it s usually super quick   For more information about Angular and SEO  we wrote an extensive tutorial on it at http   www ng-newsletter com posts serious-angular-seo html and we detailed it even more in our book ng-book  The Complete Book on AngularJS  Check it out at ng-book com

User · Answer

Things have changed quite a bit since this question was asked  There are now options to let Google index your AngularJS site  The easiest option I found was to use http   prerender io free service that will generate the crwalable pages for you and serve that to the search engines  It is supported on almost all server side web platforms  I have recently started using them and the support is excellent too    I do not have any affiliation with them  this is coming from a happy user

User · Answer

Angular s own website serves simplified content to search engines  http   docs angularjs org   escaped fragment   tutorial step 09  Say your Angular app is consuming a Node js Express-driven JSON api  like  api path to resource  Perhaps you could redirect any requests with   escaped fragment  to  api path to resource html  and use content negotiation to render an HTML template of the content  rather than return the JSON data   The only thing is  your Angular routes would need to match 1 1 with your REST API   EDIT  I m realizing that this has the potential to really muddy up your REST api and I don t recommend doing it outside of very simple use-cases where it might be a natural fit   Instead  you can use an entirely different set of routes and controllers for your robot-friendly content  But then you re duplicating all of your AngularJS routes and controllers in Node Express   I ve settled on generating snapshots with a headless browser  even though I feel that s a little less-than-ideal

User · Answer

I have found an elegant solution that would cover most of your bases  I wrote about it initially here and answered another similar StackOverflow question here which references it   FYI this solution also includes hardcoded fallback tags in case Javascript isn t picked up by the crawler  I haven t explicitly outlined it  but it is worth mentioning that you should be activating HTML5 mode for proper URL support   Also note  these aren t the complete files  just the important parts of those that are relevant  If you need help writing the boilerplate for directives  services  etc  that can be found elsewhere  Anyway  here goes     app js  This is where you provide the custom metadata for each of your routes  title  description  etc     routeProvider     when               templateUrl   views homepage html          controller   HomepageCtrl          metadata               title   The Base Page Title              description   The Base Page Description              when   about            templateUrl   views about html          controller   AboutCtrl          metadata               title   The About Page Title              description   The About Page Description            metadata-service js  service   Sets the custom metadata options or use defaults as fallbacks   var self   this      Set custom options or use provided fallback  default  options self loadMetadata   function metadata      self title   document title   metadata title     Fallback Title     self description   metadata description     Fallback Description     self url   metadata url     location absUrl      self image   metadata image     fallbackimage jpg     self ogpType   metadata ogpType     website     self twitterCard   metadata twitterCard     summary large image     self twitterSite   metadata twitterSite      fallback handle          Route change handler  sets the route s defined metadata  rootScope  on   routeChangeSuccess   function  event  newRoute      self loadMetadata newRoute metadata         metaproperty js  directive   Packages the metadata service results for the view   return     restrict   A     scope        metaproperty             link  function postLink scope  element  attrs        scope default   element attr  content        scope metadata   metadataService          Watch for metadata changes and set content     scope  watch  metadata   function  newVal  oldVal          setContent newVal          true           Set the content attribute with new metadataService value or back to the default     function setContent metadata          var content   metadata scope metaproperty     scope default        element attr  content   content              setContent scope metadata            index html  Complete with the hardcoded fallback tags mentioned earlier  for crawlers that can t pick up any Javascript    lt head gt     lt title gt Fallback Title lt  title gt     lt meta name  description  metaproperty  description  content  Fallback Description  gt      lt  -- Open Graph Protocol Tags -- gt     lt meta property  og url  content  fallbackurl com  metaproperty  url  gt     lt meta property  og title  content  Fallback Title  metaproperty  title  gt     lt meta property  og description  content  Fallback Description  metaproperty  description  gt     lt meta property  og type  content  website  metaproperty  ogpType  gt     lt meta property  og image  content  fallbackimage jpg  metaproperty  image  gt      lt  -- Twitter Card Tags -- gt     lt meta name  twitter card  content  summary large image  metaproperty  twitterCard  gt     lt meta name  twitter title  content  Fallback Title  metaproperty  title  gt     lt meta name  twitter description  content  Fallback Description  metaproperty  description  gt     lt meta name  twitter site  content   fallback handle  metaproperty  twitterSite  gt     lt meta name  twitter image src  content  fallbackimage jpg  metaproperty  image  gt   lt  head gt    This should help dramatically with most search engine use cases  If you want fully dynamic rendering for social network crawlers  which are iffy on Javascript support   you ll still have to use one of the pre-rendering services mentioned in some of the other answers   Hope this helps

User · Answer

A good practice can be found here   http   scotch io tutorials javascript angularjs-seo-with-prerender-io  escaped fragment  tag

User · Answer

With Angular Universal  you can generate landing pages for the app that look like the complete app and then load your Angular app behind it  Angular Universal generates pure HTML means no-javascript pages in server-side and serve them to users without delaying  So you can deal with any crawler  bot and user  who already have low cpu and network speed  Then you can redirect them by links buttons to your actual angular app that already loaded behind it  This solution is recommended by official site  -More info about SEO and Angular Universal-

User · Answer

The crawlers do not need a rich featured pretty styled gui  they only want to see the content  so you do not need to give them a snapshot of a page that has been built for humans   My solution  to give the crawler what the crawler wants   You must think of what do the crawler want  and give him only that   TIP don t mess with the back  Just add a little server-sided frontview using the same API

User · Answer

Use something like PreRender  it makes static pages of your site so search engines can index it   Here you can find out for what platforms it is available  https   prerender io documentation install-middleware asp-net

[html] How do search engines deal with AngularJS applications?

Examples related to html

Examples related to angularjs

Examples related to seo

Examples related to search-engine

Examples related to google-search