@Neil's answer is one (perfectly valid!) way of doing it, but you can also simply call matplotlib.use('Agg') before importing matplotlib.pyplot, and then continue as normal.
E.g.
import matplotlib as mpl
mpl.use('Agg')
import matplotlib.pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(range(10))
fig.savefig('temp.png')
You don't have to use the Agg backend, as well. The pdf, ps, svg, agg, cairo, and gdk backends can all be used without an X-server. However, only the Agg backend will be built by default (I think?), so there's a good chance that the other backends may not be enabled on your particular install.
Alternately, you can just set the backend parameter in your .matplotlibrc file to automatically have matplotlib.pyplot use the given renderer.