Yes, a Jew by birth is still considered a Jew if they reject the religion. The only way that they are no longer classified as a Jew is if they actively join a non-Jewish religion. In this case, they would obviously be ethnically Jewish. Their DNA doesn't change. But they would lose all rights within the Jewish community, and that includes any rights that the State of Israel extends to Jews, including the Right of Return.
Although they are obviously still an ethnic Jew, for all intents and purposes, they are treated as whatever they have decided to be (a Hindu, or Christian, or Muslim, or whatever). Essentially, they are treated as a non-Jew. And the State of Israel treats them as such for legal purposes.
On the other hand, if a Jew by Birth becomes an atheist, he remains a Jew ethnically and culturally, and continues to be so regarded within the Jewish community. Although the religious authorities may consider him a "bad" Jew, he IS still a Jew in both the ethnic and cultural sense of the word, and this is also recognised for legal purposes by the State of Israel, where he retains his Jewish identity, and is still allowed the Right of Return, and any other rights that Israel extends to Jews.
Its all rather complicated, and gets into the VERY complicated question of "who is a Jew?". And trying to answer that question thoroughly would keep us here for about the next six weeks. If you like, I can attempt to discuss that with you. Just say the word. But I daresay my wife will be coming home from work shortly, so I must buzz off for a bit. I'll be on a bit longer, and then be back later or tomorrow.