Einstein was in a public debate with Niels Bohr when said that. Einstein was charismatic and very good at public speaking, unlike Bohr who was shy and introverted. Bohr's reply did not receive the same press - "Don't tell god what to do."
It is annoying to hear Einstein being quoted to support religiosity and all it shows is the ignorance of the garden variety theist.
Einstein was a German Jew. He lived in Germany during WWI and left for the USA in 1930. He consistently and unambiguously denied believing in personal gods who answered prayers or involved themselves in human affairs - exactly the sort of god common to religious theists claiming that Einstein was one of them.
It seems to me that the idea of a personal God is an anthropological concept which I cannot take seriously. I also cannot imagine some will or goal outside the human sphere.... Science has been charged with undermining morality, but the charge is unjust. A man's ethical behavior should be based effectually on sympathy, education, and social ties and needs; no religious basis is necessary. Man would indeed be in a poor way if he had to be restrained by fear of punishment and hope of reward after death.
Common to all these types is the anthropomorphic character of their conception of God. In general, only individuals of exceptional endowments, and exceptionally high-minded communities, rise to any considerable extent above this level. But there is a third stage of religious experience which belongs to all of them, even though it is rarely found in a pure form: I shall call it cosmic religious feeling. It is very difficult to elucidate this feeling to anyone who is entirely without it, especially as there is no anthropomorphic conception of God corresponding to it.
- Albert Einstein, "Religion and Science," New York Times Magazine, November 9, 1930
'I believe in Spinoza's God who reveals himself in the orderly harmony of what exists, not in a God who concerns himself with the fates and actions of human beings.'
- Albert Einstein, responding to Rabbi Herbert Goldstein's question "Do you believe in God?" quoted in: Has Science Found God?, by Victor J Stenger
It was, of course, a lie what you read about my religious convictions, a lie which is being systematically repeated. I do not believe in a personal God and I have never denied this but have expressed it clearly. If something is in me which can be called religious then it is the unbounded admiration for the structure of the world so far as our science can reveal it.
- Albert Einstein, letter to an atheist (1954), quoted in Albert Einstein: The Human Side, edited by Helen Dukas & Banesh Hoffman
During the youthful period of mankind's spiritual evolution, human fantasy created gods in man's own image who, by the operations of their will were supposed to determine, or at any rate influence, the phenomenal world.
- Albert Einstein, quoted in: 2000 Years of Disbelief, James Haught
I have repeatedly said that in my opinion the idea of a personal God is a childlike one. You may call me an agnostic, but I do not share the crusading spirit of the professional atheist whose fervor is mostly due to a painful act of liberation from the fetters of religious indoctrination received in youth. I prefer an attitude of humility corresponding to the weakness of our intellectual understanding of nature and of our own being.
- Albert Einstein to Guy H. Raner Jr., Sept. 28, 1949, quoted by Michael R. Gilmore in Skeptic magazine, Vol. 5, No. 2
I cannot conceive of a personal God who would directly influence the actions of individuals, or would directly sit in judgment on creatures of his own creation. I cannot do this in spite of the fact that mechanistic causality has, to a certain extent, been placed in doubt by modern science. My religiosity consists in a humble admiration of the infinitely superior spirit that reveals itself in the little that we, with our weak and transitory understanding, can comprehend of reality. Morality is of the highest importance -- but for us, not for God.
- Albert Einstein, from Albert Einstein: The Human Side, edited by Helen Dukas & Banesh Hoffman
Nobody, certainly, will deny that the idea of the existence of an omnipotent, just, and omnibeneficent personal God is able to accord man solace, help, and guidance; also, by virtue of its simplicity it is accessible to the most undeveloped mind. But, on the other hand, there are decisive weaknesses attached to this idea in itself, which have been painfully felt since the beginning of history. ...
- Albert Einstein, Science and Religion (1941)
The more a man is imbued with the ordered regularity of all events the firmer becomes his conviction that there is no room left by the side of this ordered regularity for causes of a different nature. For him neither the rule of human nor the rule of divine will exist as an independent cause of natural events. ...
- Albert Einstein, Science and Religion (1941)
It is generally regarded that AE was a pantheist and he self described as having a 'cosmic religious sense'.