It all depends on the amount of traffic that's hitting your site. If people can't get to it because it's timing out, then your SEO will be bad because no one can load your site. Depends entirely on how it's performing for you now, on your current server, site optimization, and number of clicks.
* Please note that I offer free advice on this forum. I DO NOT PROVIDE ANSWERS VIA PM. If you would like personalized help, you can see my pricing on my services page. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *