His mother was not yet married when He was born -- and while we know about the Holy Spirit and God's plan, the overwhelming majority of people in Mary's community (and family?) would have assumed the more obvious: Joseph or "some other man."
Jesus was born when Mary and Joseph were away from home -- in a borrowed space where the best option for a cradle was a manger.
Before he was very old at all, Jesus traveled even farther away from home into Egypt -- a fugitive -- and lived there with his earthly family as refugees for a season.
We know that Joseph was still with the family when Jesus was 12 because we read about that incident at the temple, but there is no mention of him when Jesus is 30ish. We can assume that Jesus knew the sorrow of death in a very personal way.
While we don't know much about his childhood, we know enough to know that life was not easy for this boy growing up in Bethlehem-Egypt-Nazareth, a Jew in a land occupied by the Romans. And, unlike the rest of us who can blurt out "I didn't choose to be born into this family" or "I didn't choose to be born a citizen of this country" or "I didn't ask to be tall/short/smart/athletic/artistic/_______ or whatever else thing we are finding momentarily oppressive or restrictive or embarrassing -- Jesus did choose.
Think of yourselves the way Christ Jesus thought of himself. He had equal status with God but didn't think so much of himself that he had to cling to the advantages of that status no matter what. Not at all. When the time came, he set aside the privileges of deity and took on the status of a slave, became human! Having become human, he stayed human. It was an incredibly humbling process. He didn't claim special privileges. Instead, he lived a selfless, obedient life and then died a selfless, obedient death—and the worst kind of death at that—a crucifixion. (Philippians 2:5-8 The Message)