Over the years, since I was first introduced to this "pattern", this happens once a year or so. It would happen more, if I used it more. This recent case was a case where I had broke my personal rule a while back, and it caught up with me. Again.
It's not a totally evil pattern, I see its value (and use it) for things like class var singletons.
The problem I have with it is one of Predictability and Responsibility. It seems like an encapsulated thing to do. You 'grow' the state of an object as different callers call upon for it services. So the state comes into being only as its needed. For simple patterns, this works fine:
^foo ifNil: [foo := 42]
^bar ifNil: [bar := 18]
No problems. It's hard to imagine what will go wrong at this point. What often happens though, is that objects evolve over time. Different people come along and maintain them. They do so, not with the whole object in mind, but just looking at one view. So someone discovers that foo and bar actually have some interplay. And we end up with
foo ifNil: [self useConsistentFooBars ifTrue: [foo := self bar * 10]].
And then someone later does something like
bar ifNil: [self useConsistentFooBars ifTrue: [bar := self foo / 10]].
The thing is, you might get away with this for a while. It's quite possible that when this was done, all uses of the object were using a setter to set foo before either accessor is invoked. So things Just Work(tm). Until later when someone changes the order of the way the object is being talked to.
In short, as the nature of lazy initializers grows in complexity, the odds rise that the object has hidden expectations about how it has to be interfaced with. And that is anything but encapsulated. Now you have the internal implementation of the object leaking out in hard to see or document ways.