This is a rather broad question, and there are too many hypotheses to mention in this answer as Google has over 250 variables that it considers in its algorithm. However, there are a number of high-quality SEO companies/ agencies, free resources, Google’s own SEO guide, and free tools such as Google Webmaster Tools and Woorank that can help and support you in your quest to solve this problem.
The one thing to remember is that Google has not manually penalised your website, as it is solely based on one single highly complex algorithm to determine search engine rankings. Some things to consider are:
These are some broad questions that might help you solve the problem, however, it is probably advisable to hire a SEO agency as they are specialists in implementing strategies to make sure that Google loves you in the short, medium, and long-term.