File.OpenBinaryDirect may cause exception when you are using Oauth accestoken Explained in This Article
Code should be written as below to avoid exceptions
Uri filename = new Uri(filepath);
string server = filename.AbsoluteUri.Replace(filename.AbsolutePath,
"");
string serverrelative = filename.AbsolutePath;
Microsoft.SharePoint.Client.File file =
this.ClientContext.Web.GetFileByServerRelativeUrl(serverrelative);
this.ClientContext.Load(file);
ClientResult<Stream> streamResult = file.OpenBinaryStream();
this.ClientContext.ExecuteQuery();
return streamResult.Value;