Charlie, you will be fine. You will be doing what you are good at and what you truly know. I think your client base will grow when they learn how you do things and what kind of a person you are. As many of the guys on here know, you are definitely top shelf!
A buddy of mine did the same thing and started out small by himself and earns comfortable living. He isn't making himself rich, but his health and well-being doing what he wants and not working for jackasses is worth much more than some extra money.
If times get slow, you can keep an eye out for cars on craigslist that people don't want to fix or fix up and sell cheap. You can fix in spare time if you have an open day or open few hours, then turn around and sell for a profit.
Things happen for a reason, and oftentimes people get comfortable, even if unhappy. When you take chances and work hard, I think it pays off. You will be good, and I will say a prayer for your success brother!