The following is quoted from the Chofetz Chaim when discussing the importance of not involving oneself in idle conversation in shul: (Chofetz Chaim, Introduction, Positive Commandments, #6)
"Now that we are on the topic of the severe prohibition of idle conversation in the shul, I have seen it appropriate to mention the greatness of the stumbling-block that is created through these conversations. This being that it is common in our numerous sins that prior to the Reading of The Torah one congregant begins to relate to his friends of his anecdotes which are mixed in with lashon hara and rechilus from their beginning to their end. Before finishing the anecdote, they begin to read The Torah and the yetzer hara entices him to not pause in the middle of his story, so he finishes the lashon hara during the Reading of The Torah. Many times this person is one of the distinguished people who sits in the eastern part of the shul, his sin is witnessed by the public. By acting in this manner, he desecrates Hashem's Name publicly, that being in front of ten Jews, his sin being much greater than a typical desecration of Hashem's Name, as is explained in "Sefer Hamitzovs" of the Rambam (Prohibiton 63), see there. He violates the prohibition of "Do not desecrate My Holy Name..." (Vayikra: 22; 32)
By speaking idle conversation in shul as described above, the person violates the following prohibitions:
1. The prohibition of lashon hara and richilus which themselves include many prohibitions and positive commandments.
2. He has violated the prohibition of "Do not desecrate My Holy Name..." in front of ten Jews.
3. He missed out on the mitzvah of krias haTorah, which is the case even if he missed hearing one verse or even one word, it being a severe sin, his sin being too great to bear, for even leaves in the midst of the reading from the Torah is referred to in the pasuk, "and those who abandon Hashem will be destroyed" (Yeshaya: 1, 28) (Berachos, 8a - Taught in the name of Rav Ammi).
A number of times idle conversation takes place in the shul on Shabbos, the person's sin being much greater than during the weekdays, as is explained in a number of holy sifarim. It happens regularly that those who are accustomed to speak in the shul or in the beis medrash, will finish their anecdotes even during the recitation of Kaddish, and prevents himself from saying "Amen Yihei Shimei Rabba...", its' recitation being very sublime according to the gemara, as Chazal have said (Shabbos, 119b) that even if someone has a bit of Apikorsus, he is forgiven for it on account of reciting "Amen Yihei Shimei Rabba..." Even if the talker will respond by reciting "Yihei Shimei Rabba...", if it's recited too late (more than approx. 2 seconds after he can recite it), it is an Amen Yisomah (orphaned Amen), and Heaven Forbid that his children will be orphaned due to this, for this curse can befall one even if he pushed off the Amen out of laziness, as is explicit in the poskim (Orach Chaim, siman 224, sif 8).
Aside from the above, the talkers in our time transgress a fourth prohibition, that being idle conversation in the shul and in the beis medrash, it being a great prohibition that is elucidated in the Shulchan Aruch, as is written above... woe to those who speak and those who listen, as the Gra already wrote in his holy letter, "Alim Litirufah", that for every single word the person must descend a great deal into Shiole. It is impossible to estimate the magnitude of the suffering and the troubles that the person experiences for every single word, not one word being lost, see there for more."
I think the words of Chazal and of the Chofetz Chaim speak for themselves. The best thing to do is to only speak when you know it's acceptable.