Found a great solution which would help in any such situation, given you just need a runnable jar, which you do in most cases. If your application is running in Intellij Idea follow these steps: 1) Go to module settings and then artifacts, and add a jar and define main class 2) Then go to Build in the menu and click "build artifact" and you get the jar.
This worked even when I changed the source folder and used scala instead of java.