You can use both with relatively few issues. Twitter Bootstrap uses jQuery 1.7.1 (as of this writing), and I can't think of any reasons why you cannot integrate additional Jquery UI components into your HTML templates.
I've been using a combination of HTML5 Boilerplate & Twitter Bootstrap built at Initializr.com. This combines two awesome starter templates into one great starter project. Check out the details at http://html5boilerplate.com/ and http://www.initializr.com/ Or to get started right away, go to http://www.initializr.com/, click the "Bootstrap 2" button, and click "Download It". This will give you all the js and css you need to get started.
And don't be scared off by HTML5 and CSS3. Initializr and HTML5 Boilerplate include polyfills and IE specific code that will allow all features to work in IE 6, 7 8, and 9.
The use of LESS in Twitter Bootstrap is also optional. They use LESS to compile all the CSS that is used by Bootstrap, but if you just want to override or add your own styles, they provide an empty css file for that purpose.
There is also a blank js file (script.js) for you to add custom code. This is where you would add your handlers or selectors for additional jQueryUI components.