An empire dominates other territories and their inhabitants, who are seen as inferior to, or subjugated by, the dominating nation and importantly are exploited for their resources. A Caliphate assimilates territories as equals to itself, with the purpose of removing oppression and tyranny and implementing justice.
The Caliphate did not feature the domination of one race, tribe or capital city over others, neither was citizenship conditional on long military service. All were considered equal citizens, with exceptions made only for non-Muslim males not being obligated to join the Caliphate's military reserves and were expected to pay a tax for the military upkeep of the frontiers instead. If they were willing to join their fellow Muslim male citizens in the Military reserves, their tax duties were waived.
Historically territories like Cordoba rivalled Baghdad, whilst Samarkand rivalled Cairo in terms of wealth, learning and technology. It is why, unlike an empire, the capital cities often changed location in the Caliphate, Constantinople going from being a conquered city to becoming the capital of the Muslim world; likewise the capital historically moved from Medina, to Damascus, later to Baghdad and finally Istanbul.
The Islamic lands were all semi-autonomous and prospered as such. The Caliphate was no more an empire than the European Union is today.
Comparing the British occupation of Sri Lanka with Al-Andalus illustrates this difference well. The Sri Lankan people were stripped of the ownership of their lands and reduced to poverty by the British Empire's Wastelands Ordinance law. The British Empire then instituted industrial scale coffee, tea and rubber cultivation from these lands, using imported Tamil forced laborers, with resources going to Britain and profits to wealthy British 'landowners' of these lands.
The Muslim rule of Al-Andalus (Iberia) was very different. At first, Muslims were mostly confined to manning frontier barracks. With the exception of new technology, learning and trade, the non-Muslim population was almost completely unaffected. With the rise of wealth, improvements to agriculture, architecture and Muslim migration and conversions to Islam, the populations became more urbanised. Although trade increased along the Mediterranean, Al-Andalus's agriculture wasn't turned into a cash crop for the Caliphs of Damascus or Baghdad, nor were non-Muslims deprived of lands or forced into servitude. Al-Andalus over time came to rival Islamic Syria, Iraq and Egypt in security, prosperity and development. All these provinces were previously conquered territories themselves.