It turned out that my problem was that I was using a load balancer to handle the SSL, which then sent it over http to the actual server, which then complained.
Description of a fix is here: http://blog.hackedbrain.com/2006/09/26/how-to-ssl-passthrough-with-wcf-or-transportwithmessagecredential-over-plain-http/
Edit: I fixed my problem, which was slightly different, after talking to microsoft support.
My silverlight app had its endpoint address in code going over https to the load balancer. The load balancer then changed the endpoint address to http and to point to the actual server that it was going to. So on each server's web config I added a listenUri for the endpoint that was http instead of https
<endpoint address="" listenUri="http://[LOAD_BALANCER_ADDRESS]" ... />