Wasn't sure if you meant which Oracle users can read\write with the directory or the correlation of the permissions between Oracle Directory Object and the underlying Operating System Directory.
As DCookie has covered the Oracle side of the fence, the following is taken from the Oracle documentation found here.
Privileges granted for the directory are created independently of the permissions defined for the operating system directory, and the two may or may not correspond exactly. For example, an error occurs if sample user hr is granted READ privilege on the directory object but the corresponding operating system directory does not have READ permission defined for Oracle Database processes.