No, you are not missing anything. I can live anywhere. But if my mother tongue is a language other than the one commonly used in the street, then I am native to that tongue, as well as (likely) the one used in the street. I had a lot of friends whose mother tongues were both English and Spanish. Wikipedia defines a native speaker of Esperanto as a person who has at least one parent who teaches them Esperanto from birth. They go on to say that, for whatever reason, this is almost always the father.
QUOTE: "Native Esperanto speakers (in Esperanto denaskuloj or denaskaj esperantistoj) are people who have acquired Esperanto as one of their native languages. As of 1996, there were 350 or so attested cases of native Esperanto speakers,[1] though estimates from associations indicate that there are currently around 1,000 Esperanto-speaking families, involving perhaps 2,000 children.[2] In all known cases, speakers are natively bilingual, raised in both Esperanto and either the local national language or the native language of their parents. In all but a handful of cases, it was the father who used Esperanto with the child. In the majority of such families, the parents had the same native language, though in many the parents had different native languages, and only Esperanto in common.[1][3]" Wikipedia
I learned Spanish starting in the Ninth Grade, but acquired virtual native fluency in it. And I used it as a mother tongue in Latin America for almost three years. So, although I don't define myself as a native speaker, I do say that I have native fluency in it, which is nearly the same thing. But it is a bit different than learning to speak with a language, I will acknowledge that. I learned to speak with English only. I also learned Latin, but I lost most of it, which is why I am getting it back now, though prolonged study. Its coming back, but its not easy.