Check what the output on the adaptors are. If they are both, say, 5V 1A DC then you should be fine interchanging them.

Im no expert but this suggests to me that either the transmitters of your devices are not very powerful or the receiver of your hotspot is not recieving the signal well. Either way you are likely dropping upload packets so they are having to be repeated causing speed loss.

Does it vary within line of sight or are there walls in the way?