If you're going to build complex user interfaces in the browser then you will probably find yourself eventually inventing most of the pieces that make up frameworks like Backbone.js and Sammy.js. So the question is, are you building something complicated enough in the browser to merit using it (so you don't end up inventing the same thing yourself).
If what you plan to build is something where the UI regularly changes how it displays but does not go to the server to get entire new pages then you probably need something like Backbone.js or Sammy.js. The cardinal example of something like that is Google's GMail. If you've ever used it you'll notice that it downloads one big chunk of HTML, CSS, and JavaScript when you first log in and then after that everything happens in the background. It can move between reading an email and processing the inbox and searching and back through all of them again without ever asking for a whole new page to be rendered.
It's that kind of app that these frameworks excel at making easier to develop. Without them you'll either end up glomming together a diverse set of individual libraries to get parts of the functionality (for example, jQuery BBQ for history management, Events.js for events, etc.) or you'll end up building everything yourself and having to maintain and test everything yourself as well. Contrast that with something like Backbone.js that has thousands of people watching it on Github, hundreds of forks where people may be working on it, and hundreds of questions already asked and answered here on Stack Overflow.
But none of it is of any importance if what you plan to build is not complicated enough to be worth the learning curve associated with a framework. If you're still building PHP, Java, or something else sites where the back end server is still doing all the heavy lifting of building the web pages upon request by the user and JavaScript/jQuery is just icing upon that process, you aren't going to need or are not yet ready for Backbone.js.