If I have to pull ball out of a bag and I have a $4\%$ chance of picking the winning ball. I have $3$ attempts to pick out the winning ball (each time the odds are $4\%$ of success as the loosing ball is returned) What is my overall chance of picking out a winning ball? I would have believed it still to be $4\%$ overall but I am being told differently?
You have a $96\%$ chance of losing each attempt, correct?
The chances of losing three attempts in a row is therefore $96\%\cdot96\%\cdot96\% = 88.4736\%$
So your chances of winning at least once is $11.5264\%$
And it kind of makes sense, doesn't it? The probability is just under $12\%$, and you have three attempts each at $4\%$.
The real concept question here is, why isn't it $12\%$?
Your chance of failing is $0.96$ each time. Since the tries are independent, the chance of failing three times is $.96^3=0.884736$, so your probability of success $=0.115264$.
Often in calculating probabilities, it is sometimes easier to calculate the probability of the 'opposite', the technical term being the complement. Because if something happens with probability $p$, then it does not happen with probability $1-p$, e.g. if something happens with probability $0.40$ ($40\%$) then it does not happen with probability $1-0.40=0.60$ ($60\%$).
The 'opposite' (complement) of winning at least once is never winning at all. The probability of not picking the winning ball the first time is $1-0.04=0.96$, i.e. $96\%$. But you also want this to happen the second time and the third time. So you do not win with probability $$ 0.96 \cdot 0.96 \cdot 0.96= 0.884736, $$ i.e. $88.4736\%$. But then this is the probability that you do not win so that the probability that you win is $1-0.884736= 0.115264$, or $11.5264\%$.
You can pick a winning ball in one of 3 disjoint events:
Therefore, sum of the probabilities of these 3 disjoint events is $0.04 + 0.96*0.04 + 0.96^2 * 0.04$