It can be observed that Matthew is wealthy enough to own a large house which can accommodate many people seated for a meal. I find it interesting that the Jewish religious leaders either had free access into peoples' homes or they had plenty of spies informing them of issues.
Matthew had yielded himself to Jesus and naturally he wanted his friends to know Him too, so what better than a meal! His friends were, inevitably, like him, as no respectable person would meet with tax collectors. In one sense, these were outcasts of society, though wealthy. It is important to recognise that those who faithfully followed Jewish Law did not want to spend time with such people and had no compunction to feel they should: this is one of the many ways Jesus transformed the moral, spiritual and ethical landscape when He ministered. Jesus defied the orthodox conventions of His day.
It is also vital to recognise that 'sinners' did not necessarily indicate moral turpitude. It could also mean in Jesus' day someone who did not observe the scribal Law. In a Jew's eyes, the person who committed adultery and the person who ate pork or didn't wash his hands the correct number of times were both sinners! Obedience was often impossible for those who had to work to live, for scribal law brought so many infinitesimal matters to the table.
When I lead times of Communion around the Lord's table, I often declare everyone present to be a sinner and it is only when we acknowledge ourselves to be fallen creatures that God can come in and transform us. Those who looked down in judgement on what Jesus did were not in the right place spiritually to accept their need of His saving grace. Their attitude was one of contempt and fear for sinners: ideally, they would have them vanish from the earth. Does your heart go out to those who are in darkness? Are you full of compassion for those who have no leader they can look up to and whose lives are driven by fear and anxiety?