As already stated, you can get information about your Releases via the API.
For those using WordPress, I developed this plugin: GitHub Release Downloads. It allows you to get the download count, links and more information for releases of GitHub repositories.
To address the original question, the shortcode [grd_count user="User" repo="MyRepo"]
will return the number of downloads for a repository. This number corresponds to the sum of all download count values of all releases for one GitHub repository.
Example: