For linux/unix, I can suggest unix package.
To increase the memory limit in linux:
install.packages("unix")
library(unix)
rlimit_as(1e12) #increases to ~12GB
You can also check the memory with this:
rlimit_all()
for detailed information: https://rdrr.io/cran/unix/man/rlimit.html
also you can find further info here: limiting memory usage in R under linux