Usually you would have something like "ADEVICE?plughw:1,0" when running on the Raspberry Pi. ?I don't see ADEVICE in there which would result in using the "default" device.
Did you make sure that PulseAudio was removed? ?In my experience PulseAudio never worked right on the RPi and my recommendation is to remove it. ? (Note: ?It is fine on larger x86 based systems.)
?? ?Currently section 1.3.
The other thing I would try is running the "top" command in another window? ?What is using a significant amount of CPU time? ?How much idle time is left over?