The simplest approach is to use a multitouch JavaScript library like Hammer.js. Then you can write code like:
canvas
.hammer({prevent_default: true})
.bind('doubletap', function(e) { // And double click
// Zoom-in
})
.bind('dragstart', function(e) { // And mousedown
// Get ready to drag
})
.bind('drag', function(e) { // And mousemove when mousedown
// Pan the image
})
.bind('dragend', function(e) { // And mouseup
// Finish the drag
});
And you can keep going. It supports tap, double tap, swipe, hold, transform (i.e., pinch) and drag. The touch events also fire when equivalent mouse actions happen, so you don't need to write two sets of event handlers. Oh, and you need the jQuery plugin if you want to be able to write in the jQueryish way as I did.