I have been researching this for a few days, following endless leads that got close, but not quite, to what I was after.
My need is for a simple list of 'symbol, sector, industry'. I'm working in Java and don't want to use any platform native code.
It seems that most other data, like quotes, etc., is readily available.
Finally, followed a suggestion to look at 'finviz.com'. Looks like just the ticket. Try using the following:
http://finviz.com/export.ashx?v=111&t=aapl,cat&o=ticker This comes back as lines, csv style, with a header row, ordered by ticker symbol. You can keep adding tickers. In code, you can read the stream. Or you can let the browser ask you whether to open or save the file.
http://finviz.com/export.ashx?v=111&&o=ticker Same csv style, but pulls all available symbols (a lot, across global exchanges)
Replace 'export' with 'screener' and the data will show up in the browser.
There are many more options you can use, one for every screener element on the site.
So far, this is the most powerful and convenient programmatic way to get the few pieces of data I couldn't otherwise seem to easily get. And, it looks like this site could well be a single source for most of what you might need other than real- or near-real-time quotes.