android.R.id.custom was returning null for me. I managed to get this to work in case anybody comes across the same issue,
AlertDialog.Builder builder = new AlertDialog.Builder(context)
.setTitle("My title")
.setMessage("Enter password");
final FrameLayout frameView = new FrameLayout(context);
builder.setView(frameView);
final AlertDialog alertDialog = builder.create();
LayoutInflater inflater = alertDialog.getLayoutInflater();
View dialoglayout = inflater.inflate(R.layout.simple_password, frameView);
alertDialog.show();
For reference, R.layout.simple_password is :
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:orientation="vertical"
android:layout_width="match_parent"
android:layout_height="match_parent">
<EditText
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:id="@+id/password_edit_view"
android:inputType="textPassword"/>
<CheckBox
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="@string/show_password"
android:id="@+id/show_password_checkbox"
android:layout_gravity="left|center_vertical"
android:checked="false"/>
</LinearLayout>