Babies in Medical School

My DH and I had our first daughter just a few months after I completed my PhD (no easy feat to finish while super pregnant and living alone as DH had moved ahead to new job).  At the time I was going on round 2 of medical school applications and in the meantime managed to land a full-time faculty position at a technical college.  I'm not sure if its inherent to technical school college that generally hires "experts in the field" (people who have decades of experience, ie as a RN, CNA, welder, etc) and then come and teach, but I found I was a super minority as a mother with a small child.  For example, I got THREE WEEKS of maternity leave, and they actually had the gall to call me after one week to get a firm date of my return.  I won't even go into the very dark thoughts I had, one week postpartum and staring down at my 1-week daughter...but I digress.  As things progressed over the next 6 months, I got accepted to multiple medical schools, continued to hate most aspects of my job more and more (minus the teaching part which somehow was a minor part, and my paycheck which was the nicest I had after spending the last 2 decades of my life in school), and started having thoughts about child #2.
The decision is "mostly" made that I will quit said current and hated job and start medical school in August.  But I am terrified by the leap into the unknown.  I won't have a paycheck, I won't be an "adult" anymore (I'll be a student again), I have no idea if I'll have tons or hardly any time to spend with my family, and we are even going with the crazy plan of having me "commute" 2hrs each way so DH can keep his paycheck and DLO (darling little one) can stay at her amazing daycare that took me three months, three tries, and millions of panic attacks to find.
So, am I crazy?  I want three more kids, but I also want to do well (enough) in medical school, and we're trying to keep our current home.  It all sounds crazy and unrealistic to me!
Part of me thinks that having more children in my current job would be just as hard if not harder than having them in medical school.  I only get 3 weeks paid maternity leave here and at least in medical school I'll have summer, winter, and spring breaks plus some open blocks for studying, interviewing, and vacation...a luxury not found in my current job.  I also think that while medical school is HARD, is it any harder than a very demanding, more than full-time job?  Or getting a biomedical PhD in 4 years (by working 60+ hours a week)?  Some people swear it is, others that it isn't.  It just kills me to not know for sure what it will be like.  It's tough to leave what is unknown, even if unliked for something that could potentially be even worse.  And at least financially, I know that will be the case.  There's is something comforting about being financially stable!  Eventually we'll get more than back there but those intervening years will be tough.
So if I could have my cake and eat it too and pregnancies went perfectly and I did super well in all my classes with the normal amount of effort: baby #2 between M1 and M2, baby #3 during M4, and baby #4 during the end of residency.  Yup, I'm crazy.  But I do know for sure that nothing is more important to me than family so one way or another, we'll have to make it work!