In the Quran, the term nabi (نَبِي) is typically translated as prophet, while rasool (رَسُول) is translated as messenger. As stated in previous articles, a prophet is given (āta / آتَيْ) scripture (kitab / كِتَاب), which are direct verbatim words from God. While a messenger sends and confirms an existing message from God.
Another way to think of this is that when a person is receiving direct words from God they are acting in the capacity of a prophet, but when they are sending the message to others they are acting in the capacity of a messenger. The verse that best articulates this distinction can be found in 3:81.
3:81 GOD took a covenant from the prophets, saying, “I will give you the scripture and wisdom. Afterwards, a messenger will come to confirm all existing scriptures. You shall believe in him and support him.” He said, “Do you agree with this, and pledge to fulfill this covenant?” They said, “We agree.” He said, “You have thus borne witness, and I bear witness along with you.”
وَإِذْ أَخَذَ اللَّهُ مِيثَاقَ النَّبِيِّينَ لَمَا آتَيْتُكُمْ مِنْ كِتَابٍ وَحِكْمَةٍ ثُمَّ جَاءَكُمْ رَسُولٌ مُصَدِّقٌ لِمَا مَعَكُمْ لَتُؤْمِنُنَّ بِهِ وَلَتَنْصُرُنَّهُ قَالَ أَأَقْرَرْتُمْ وَأَخَذْتُمْ عَلَىٰ ذَٰلِكُمْ إِصْرِي قَالُوا أَقْرَرْنَا قَالَ فَاشْهَدُوا وَأَنَا مَعَكُمْ مِنَ الشَّاهِدِينَ
If we look at the roots of these words their meanings become even more clear. For instance, the word (rasool) رَسُول comes from the root ر س ل which has the derivative meaning to send, like in the word أَرْسَلَ (arsala).
Nabi, on the other hand, comes from the root ن ب ا, which has the derivative meaning to inform or news, like in the words أَنْبَاءِ / anbāi and نَبِّئْ / nabbi. This signifies that nabi is the recipient of the new information from God.