There is, alas, another use of "upstream" that the other answers here are not getting at, namely to refer to the parent-child relationship of commits within a repo. Scott Chacon in the Pro Git book is particularly prone to this, and the results are unfortunate. Do not imitate this way of speaking.
For example, he says of a merge resulting a fast-forward that this happens because
the commit pointed to by the branch you merged in was directly upstream of the commit you’re on
He wants to say that commit B is the only child of the only child of ... of the only child of commit A, so to merge B into A it is sufficient to move the ref A to point to commit B. Why this direction should be called "upstream" rather than "downstream", or why the geometry of such a pure straight-line graph should be described "directly upstream", is completely unclear and probably arbitrary. (The man page for git-merge does a far better job of explaining this relationship when it says that "the current branch head is an ancestor of the named commit." That is the sort of thing Chacon should have said.)
Indeed, Chacon himself appears to use "downstream" later to mean exactly the same thing, when he speaks of rewriting all child commits of a deleted commit:
You must rewrite all the commits downstream from 6df76 to fully remove this file from your Git history
Basically he seems not to have any clear idea what he means by "upstream" and "downstream" when referring to the history of commits over time. This use is informal, then, and not to be encouraged, as it is just confusing.
It is perfectly clear that every commit (except one) has at least one parent, and that parents of parents are thus ancestors; and in the other direction, commits have children and descendants. That's accepted terminology, and describes the directionality of the graph unambiguously, so that's the way to talk when you want to describe how commits relate to one another within the graph geometry of a repo. Do not use "upstream" or "downstream" loosely in this situation.
[Additional note: I've been thinking about the relationship between the first Chacon sentence I cite above and the git-merge man page, and it occurs to me that the former may be based on a misunderstanding of the latter. The man page does go on to describe a situation where the use of "upstream" is legitimate: fast-forwarding often happens when "you are tracking an upstream repository, you have committed no local changes, and now you want to update to a newer upstream revision." So perhaps Chacon used "upstream" because he saw it here in the man page. But in the man page there is a remote repository; there is no remote repository in Chacon's cited example of fast-forwarding, just a couple of locally created branches.]