I bought my xB new in '06. I upgraded the head unit but kept the OEM speakers; I didn't add an amp, as I had with several other cars, because I was very satisfied with the sound.
For more than a year I noticed the sound wasn't balanced (left—right) but I was too lazy to make the adjustment. This past month I saw small pieces of speaker surrounds coming out through the door grills. I upgraded all the speakers and the sound is awesome again—so—if your xB is older than ten years you might want to look through the grills to check the condition of the surrounds (the flexible material that affixes the cone to the speaker basket-frame).