Just do this:
require_relative 'tokenizer'
If you put this in a Ruby file that is in the same directory as tokenizer.rb, it will work fine no matter what your current working directory (CWD) is.
The other answers claim you should use require './tokenizer', but that is the wrong answer, because it will only work if you run your Ruby process in the same directory that tokenizer.rb is in. Pretty much the only reason to consider using require like that would be if you need to support Ruby 1.8, which doesn't have require_relative.
The require './tokenizer' answer might work for you today, but it unnecessarily limits the ways in which you can run your Ruby code. Tomorrow, if you want to move your files to a different directory, or just want to start your Ruby process from a different directory, you'll have to rethink all of those require statements.
Using require to access files that are on the load path is a fine thing and Ruby gems do it all the time. But you shouldn't start the argument to require with a . unless you are doing something very special and know what you are doing.
When you write code that makes assumptions about its environment, you should think carefully about what assumptions to make. In this case, there are up to three different ways to require the tokenizer file, and each makes a different assumption:
require_relative 'path/to/tokenizer': Assumes that the relative path between the two Ruby source files will stay the same.require 'path/to/tokenizer': Assumes that path/to/tokenizer is inside one of the directories on the load path ($LOAD_PATH). This generally requires extra setup, since you have to add something to the load path.require './path/to/tokenizer': Assumes that the relative path from the Ruby process's current working directory to tokenizer.rb is going to stay the same.I think that for most people and most situations, the assumptions made in options #1 and #2 are more likely to hold true over time.