In simple terms, USB audio is essentially a digital alternative to analog audio.
If you connect your keyboard to external speakers, headphones, or some audio device-- including a computer's audio connections-- using standard TRS audio cables, the analog signal can be prone to things like line noise. Digital signals tend to be cleaner and less prone to noise, although certainly not immune to it.
And if you're connecting to a computer or any other device that uses digital audio, the analog signal needs to be converted to digital, and the digital signal needs to be converted to analog, depending on whether it's going to or from the computer. Keyboards that don't have USB audio must either have their analog audio converted to digital using an external audio interface before going to the computer, or else have it converted to digital by the computer's internal sound card. Some computers have high quality sound cards, but others have cards which are not as good as what an external audio interface would typically have.
Additionally, USB signals can usually carry several audio channels at the same time, as opposed to just a single mono channel or a pair of left/right stereo channels. Indeed, a single USB connection can be used for input and output simultaneously, as opposed to needing separate analog input and analog output audio jacks.
So basically, USB audio offers a better quality signal, as well as a much less complicated connection as far as the number of jacks and cables needed. And the same USB connection that's carrying the digital audio input/output signals can simultaneously be carrying MIDI input/output signals.