In a Nutshell:
The narrations of the banquet don't detail what the prophet said. However, the hadith of the trader Afif suggest he invited them to a new worldview, based on Tawheed, an alternative way of life that would lead to the two main powers and their wealth coming under his rule.
When Allah ordered his Prophet (saw) to start calling people to Allah at the very beginning of Islam, Allah asked him to start with his clan and close relatives.
وَأَنذِرْ عَشِيرَتَكَ الْأَقْرَبِينَ
"And warn, (O Muhammad), your closest kindred." (Qur'an 26:214)
The Prophet (saw) decided to start the mission by addressing his clan collectively. He asked Ali (ra) to prepare a banquet and invite leading figures of his clan (Banu Abd al-Muttalib). Ali cooked the meal and invited about forty men. The men left before the Prophet could speak with them. He invited them again and after eating he said:
ِيا بَنِي عَبْدِ المُطَلِبْ، إنِي والله ما أَعْلَمُ شَابًا فِي العَرَبِ جاءَ قَوْمَهُ بأفْضَلَ مِمَا جِئتُكُم به، قَدْ جِئْتُكُم بخيّرِ الدُنيا والآخِرَة، وقَدْ أَمَرَنِي اللهُ تعالى أن أَدْعُوكُم إليهِ، فأيُكُم يؤازِرُنِي على هَذَا الأمر
"O Banu Abd al-Muttalib, I, indeed, don't know of any Arab youth who came to his people with a notion better than I came with.
I came to you with the goodness of this world and the Here After. Allah (swt) ordered me to call you to Him.
Who is going to support me regarding this matter. (Tabari, Tarikh at-Tabari, Vol. 2, p. 322, Ibn al-Athir, al-Kamil fi at-Tarikh, Vol. 1, p. 259)
The Prophet (saw) then asked them for someone who could provide him with protection and sustenance of his family in case he was injured.
It was narrated:
مَنْ يَضْمَنُ عَنِّي دَيْنِي وَمَوَاعِيدِي ، وَيَكُونُ مَعِي فِي الْجَنَّةِ ، وَيَكُونُ خَلِيفَتِي فِي أَهْلِي ؟ " فَعَرَضَ ذَلِكَ عَلَى أَهْلِ بَيْتِهِ ، فَقَالَ عَلِيٌّ : أَنَا ، فَقَالَ رَسُولُ اللَّهِ صَلَّى اللَّهُ عَلَيْهِ وَسَلَّمَ : "عَلِيٌّ يَقْضِي عَنِّي دَيْنِي ، وَيُنْجِزُ مَوَاعِيدِي."
"Who could guarantee my deen and my responsibilities and he would be with me in Paradise and my successor (caliph) in my family? He told that to his clan. Ali answered 'I am.' The Prophet replied, 'Ali will pay off my debts and accomplish my responsibilities.'" (Ahmed, Fadda'il as-Sahabah)
In other narrations, the Prophet (saw) said:
فَأَيُّكُمْ يُبَايِعُنِي عَلَى أَنْ يَكُونَ أَخِي وَصَاحِبِي ؟
"Who of you is going to give me the pledge to be my brother and companion?" (Musnad Ahmed, Tarikh Dimashq and Tahdhib al-Kamal)
These narrations (and many other narrations) makes more sense as the Prophet (saw) was asking for someone to protect him and his dawah and to guarantee his debt and responsibilities (Tabari, Tarikh at-Tabari, Vol. 2, p. 322) in case he was injured whilst delivering the message.
Some narrations state his clan understood the implications of such support. They objected to what the Prophet (saw) was calling for as impractical - all the Arabs and non-Arabs would oppose them and become enemies and they could not fight them all.
Ali's (ra) narration report the response of Abu Lahab:
يا بَنِي عَبْد المُطَلِب، هذِهِ واللهِ السَّوْأَةُ خُذُوا على يَدَيّهِ قَبْلَ أن يَأخُذَ على يَدَيّهِ غَيْرَكُم، فَإِنْ أسْلَمتُمُوهُ حِينَئِذٍ ذَلَلّتُم، وإن مَنَعْتُمُوهُ قُتِلْتُمْ. ... إذا قَامَت بُطُونُ قُرَيِشٍ وقامَتْ مَعَهَا العربُ فما قُوَتُنَا بِهِمْ؟ فَوَاللهِ مَا نَحْنُ عِنْدَهُم إلا أكَلَةُ رَأْسٍ
"O Banu Abd al-Muttalib, by Allah, this is a dangerous matter. So prevent him before others will, and at that time you, in humility, will give him away to them (for murder) and if you protected him you would be killed...
What if all the clans of Quraysh along with the Arabs opposed, how can you struggle with them? By Allah, we according to them are nothing but a meal." (Ibn al-Athir, al-Kamil fi at-Tarikh, Vol. 2, p. 40)
Later an incident clarifying what was the Prophet's aim was.
Afif was a trader who attended Mecca during the Hajj period to do trade with Abbas ibn Abd al-Muttalib, the uncle of the Prophet (saw), who was also a trader.
While they were sitting, they saw a man came out of a tent, face the Ka'ba and start praying. Then a woman came out and stood behind him and prayed. Then a boy came and stood to his right and started praying.
One version of the hadith states Afif questioned Abbas about this religion, as it did not seem familiar.
Abbas reported the man was Muhammad ibn Abdullah, his nephew, who believed Allah had sent him as a Messenger and the treasures of the emperors of Rome and Persia would be opened for him and his followers. Beside him was his wife Khadijah, who believed in him and followed him, and the boy was his cousin Ali ibn Abu Talib, who also believed in him. Afif later said he wished he believed that day, as he would have been the fourth in Islam, the second adult to believe.
Al-Abbas (ra) narrated he answered his question:
هَذًا مُحَمَدٌ بن عبدِ اللهِ يزْعُمُ أنّ اللهَ أرْسَلَهُ وأنّ كُنُوزَ كِِسْرى وقَيْصَرْ ستُفْتَحُ عليه... فليّتَنِي كنتُ آمَنْتُ يَوْمَئِذٍ فكُنْتُ أكُونُ رَابِعًا
"The man was Muhammad ibn Abdullah, his nephew, who believed Allah had sent him as a Messenger and the treasures of the emperors of Rome and Persia would be opened for him and his followers … Afif (ra) later said he wished he believed that day, as he would have been the fourth in Islam (after the Prophet, Khadijah and Ali)." (Ibn Ishaq, as-Seera wa al-Maghazi, pp. 137-138)
This was a very early event, most certainly early in year one, as Afif argued he would have been the fourth Muslim.
The second narration describes the prayer.
The man went into ruku' and the boy and woman went into ruku' with him. After that, he stood up and the child and woman stood up with him. Then the man fell into sujood and both of them went into sujood. Afif commented this was remarkable. Abbas asked if Afif knew who these people were, commenting this was Muhammad ibn Abdullah, his nephew; the boy was Ali ibn Abu Talib and the woman was Khadijah bint Khuwaylid, the wife of the man. 'He says your Lord, the Lord of the sky, has commanded him to do what you see them doing'. Abbas commented that 'I swear by Allah, I do not know anyone other than these three who follows this religion today'.
There are other narrations for the same situation for this narration where the Prophet either did not say the last part of the narration, or refused Ali when he raised his hand for approval as his brother (not a caliph) or a successor in his family (as later the husband of his daughter Fatima not the successor of the Umah) and in others accepted his initiative. In addition, the purpose in the hadiths for the protection and taking care of his family if he was hurt, and it has nothing to do with leadership after him.
The narrations of the banquet don't detail what the prophet said. However, the hadith of the trader Afif suggest he invited them to a new worldview, based on tawheed, an alternative way of life, both of which would lead their society to destroy two main powers of the time, Rome and Persia, with their wealth coming under his rule.