For ListView inside ScrollView use NestedScrollView it can handle this functionality very easily:
<android.support.v4.widget.NestedScrollView
android:layout_width="match_parent"
android:layout_height="match_parent">
<LinearLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical">
<android.support.v7.widget.RecyclerView
android:id="@+id/recycler_view"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:padding="5dip"/>
</LinearLayout>
</android.support.v4.widget.NestedScrollView>