This question can apply equally to various customs and traditions. I chose mezuzah because it illustrates the point. For those who don't know, mezuzah is a thin and long container that Jews put on their door. The container contains an excerpt of the Old Testament, rolled into a tube. Furthermore, religious jews who are visiting the house touch the mezuzah with their hand before entering the house.
In Israel, 99% of jewish people have the mezuzah on their door. This is true also for those who don't consider themselves religious. Their rationale is threefold:
My question is what is the relationship between tradition, custom, social convention, and religion. Is their any value in customs that identify a group of people ?
I know that Ayn Rand said that tradition should not be any limitation on man. In the case of mezuzah, is it a senseless religious thing to get rid off, or is it a good custom?
Now, apply the same question to other customs, for example, Christmas, and everything associated with it. It is customary to send post-cards and give presents on Christmas. In other words, this custom is a set of a certain associated activities which you have to do if you celebrate Christmas. It is not simply a celebration of achievement and production.
Or take Irish folk dance. Irish people are proud of their dance and they send their daughters to study the dance. They are inclined to do that, more than say non-irish people. Other people are inclined to listen/play their traditional folk music. Are such inclinations to customs justified?
Let me generalize a little more -- clothes. We are wearing certain attire based on established customs. They are very different from the clothes we wore 100 years ago. Are all the customs I mentioned by now in the same category ? When does a custom become collectivism?
One must answer the concrete case before going for the general case. Since the questioner gives a clear description of the customs regarding mezuzahs, we can look at that.
A mezuzah on your door says: "We are Jews", and a religious person will not enter your house unless you have a mezuzah for them to touch. I'll presume that said religious person wouldn't require a mezuzah of every house he enters, but only of Jewish homes. Or perhaps a religious Jew never enters non-Jewish homes. Hard to tell.
Clearly, the mezuzah is a way of showing respect for your chosen religion. To live in a neighborhood where every house has a mezuzah on it probably makes a Jew feel quite at home -- he can presume he is among friends of Judaism. For people who have been as persecuted as Jews have been, throughout history, this is quite a value. They can know where they are relatively safe -- less likely to be mugged by anti-Semites.
If, however, these Jews were to live in a society where their individual rights are well protected by a secular government, and where one's neighbors are not likely to hate one for one's chosen religion, the mezuzah's value would be more questionable. In a rights-respecting society, the mezuzah would primarily be a way of saying "I'm different from you."
Whereas a symbol stating "I'm a Jew, not a thug" might make perfect sense in a society surrounded by thugs, in a civilized society, such a symbol might be considered offensive -- a symbol of arrogant righteousness where such is not appropriate.
When armies are at war, they wear uniforms, or use IFF (Identification Friend or Foe) technology to determine who is the enemy, and who is not. A mezuzah is effectively an IFF tag for Jews.
Given the history of Judaism, I can understand Jews wanting to keep their mezuzahs. But the mezuzah is a symbol of the Jew as defensive warrior, a warrior who holds non-Jews in suspicion.
Letting one's guard down, and learning to treat one's neighbors as individuals rather than primarily as potential aggressors is a value when one is not at war.
The mezuzah is, therefore, a disvalue in a peaceful society.
Every other custom must be similarly evaluated. One must ask: what does it mean? What purpose does it serve? In what contexts is that purpose valid?
Irish dancing is a skill. Do non-Irish people sometimes learn Irish dancing? Yes. Do Irish people take offense to this? Some might. Do some Irish parents force their children to learn Irish dancing? Some do. Regardless, there's nothing wrong with Irish dancing as an art form. What's wrong is to think that a particular art form is only for a particular race of people and that all members of that race must learn that art form.
Many cultural traditions are nothing more than skills passed down from generation to generation. There's nothing wrong with skills as such, but some people value old skills for their oldness. This is the basic error of tradition as such. The oldness of something is not to be considered a source of its value.
For any given skill, there are some people who have it, and some who do not. The skill, therefore, might be seen as a mark of membership in a group. The important question, though, is, does this group encourage individuals to lose themselves in it. Is the group what is important, or does each individual in the group keep his identity?
If, as someone with a skill, you enjoy exercising that skill, even though it is an old skill, there's little that can be said against you.
If, however, you spit on those without the skill, and you force your children to learn the skill, and you despise people outside your race for learning your race's skill, you are simply a racist.
An Objectivist answer to the question of what makes a cultural custom good or bad begins with identification of the standard of good and bad (value or disvalue) and for whom that standard applies. (The standard itself is a product of identification and validation, also, not an a priori premise.)
In Objectivism, the standard of value for man is man's life qua man, as Ayn Rand explains in VOS Chapter 1 (also in Galt's Speech and the excerpts in The Ayn Rand Lexicon under the topics of "Values" and "Standard of Value"). Furthermore, the purpose of morality, as identified by Objectivism, is to serve as a guide for individuals in making their own personal choices of goals and actions by which to sustain and strengthen their own lives.
The applicability of this standard and purpose to specific concretes such as Jewish mezuzahs, Christmas celebrations, Irish folk dancing, clothing styles, etc., depends on the value that an individual derives from those respective concretes, i.e., how well they serve or undermine an individual's life. The evaluation also depends on the natue of the society in which one is living. One needs to take cognizance of situations where one can easily become a target of physical force if one declines to uphold some particular custom. It's an evaluative process that each individual must perform for himself. Philosophy cannot properly prescribe a generalized "one size fits all" kind of answer where specific concretes are concerned. The question itself explains some key issues involved regarding mezuzahs for Jews living in modern Israel. Christmas actually began as a secular celeration of earthly accomplishment, and was gradually assimilated by Christianity (and given the name "Christmas") later on, as the church realized that they could not oppose so widespread and popular a celebration effectively. Nor would I fault anyone who lives in Ireland for taking a certain pride in Irish folk dances, insofar as those dances are more secular than mystical or religious. Clothing styles, too, leave great room for many options and choices, and individuals would have wide latitude in their selection of clothing under capitalism.
Rejection of "one size fits all" should not be taken to imply that values are purely subjective and relative, however. It's basically an issue of each individual's own context, with definite, objective limits on different contexts, and a high degree of objectivity within each context. Values are contextual, like all forms of knowledge of reality. Contexts are real; they are not a denial of reality, but only an aspect and manifestation of it. Holding the context doesn't mean that value is determined by culture. Culture can affect issues of physical force and how it is used or banned, which can affect how one chooses one's own course of action; but one should always examine and judge cultural norms to the best of one's ability and act on one's objective, rational judgment to the best of one's capacities and opportunities. If one has decided to reject religion, then one certainly should not display symbols of allegiance to it, if one is living in a relatively free country that separates state and church. If one actually does uphold allegiance to religion while being fully free not to do so, that is the fundamental moral issue -- pursuing life-diminishing values.
Update: Concrete Cases
In a comment, the questioner asks the following:
If your family or relatives or friends are inviting you to a Passover dinner, would you go ? For all of them it is has religious element.
To answer this question for one's own decision in a specific instance, one would need to take cognizance of a few additional aspects of the total context, such as the following:
If you're free to express your own views in their midst, then perhaps their commitment to religious customs isn't as deeply personal and authentically religious as you might think. If a "Passover dinner" is more ceremonial and superficial than a celebration of religious edicts, then the value of your relationships with them could make your interactions with them very worthwhile for you. They might even become curious about what your personal standards are and why, although it might start out as casual or mocking but friendly interest.
Applying broad principles to concrete cases always involves a wide context of details and specifics. One cannot simply treat principles as dogma, to be adhered to the same way in all situations. Fundamental "reasonableness" (i.e., rationality) does not consist in replacing an old dogma with a new one. There are usually many different ways to act in accord with the same principle, if it's a rational principle. But a principle is what it is, and it does set limits. Exactly where those limits are in concrete cases isn't always easy or simple to sort out. An analogous situation exists in the field of concepts. Two apples may be very different from each other in size and other characteristics, yet both are apples and are very different from oranges.