Since Objectvism rejects the concept of duty, what is the concept that describes why a person should respect the rights of others and has a moral obligation to not do certain things like initiate aggression or steal or desecrate someone's private property?
That would normally be understood as an obligation or responsibility, but both of those are synonymous with duty. So what is the concept that is used in place of duty?
Rand defined duty as: "the moral necessity to perform certain actions for no reason other than obedience to some higher authority, without regard to any personal goal, motive, desire or interest."
But that is not the definition of duty. Duty is defined as something that one is expected or required to do by moral or legal obligation.
So since Ayn Rand has apparently idiosyncratically defined altruism and duty off the table, is there a concept in Objectivsm that describes why one has an obligation or responsibility to not do whatever you want, regardless of the rights and feelings of others, when it is in one's rational self interest to do so?
[Duty] would normally be understood as an obligation or responsibility, but both of those are synonymous with duty. So what is the concept that is used in place of duty?
In Objectivism, the concept that guides moral choices in lieu of duty is causality, specifically teleological causality, acting to achieve a goal (and also choosing one's goals objectively, with man's life qua man as one's standard of value). Ayn Rand discusses this at length in her article, "Causality Versus Duty," published in PWNI (Chap. 10). An extensive collection of excerpts from that article can be found in The Ayn Rand Lexicon under the topics of "Duty" and "Responsibility/Obligation." Here is a brief sampling from the latter topic:
In reality and in the Objectivist ethics, there is no such thing as “duty.” There is only choice and the full, clear recognition of a principle obscured by the notion of “duty”: the Law of Causality....
This view of causality in ethics rests on a deeper foundation that includes the nature of the concept "value" and man's life qua man as the only objective standard of value for man. Further elaboration of those points can be found in other Lexicon topics. In particular, the significance of choosing to live or not may raise many issues in modern ethical thinking, even if there is no dispute about what is needed to implement and achieve the choice to live. The "qua man" aspect of "man's life" may raise further issues, requiring elaboration of the relationship between existence or non-existence, as against life "qua man" versus "living death." I can elaborate further on these points if there is interest, but genuine interest implies studying Ayn Rand's original works as well as studying the purported arguments of Objectivism's often ignorant detractors.
Note also that Ayn Rand strongly disputed the common assumption that duty is synonymous with rational obligation. Objectivism certainly rejects "duty," as Ayn Rand understands that term (from the perspective of philosophy in general and Kantianism in particular), but she upholds the concept of rational obligation or responsibility (based on causality).
"Duty," in the sense of unquestioning obedience to social or mystical authority, presupposes that "bad consequences" of some kind will be imposed on violators. Otherwise, "duty" would have little motivational effect. "Causality" involves "bad consequences for malfeasance," too, but the consequences are imposed primarily by reality itself, not by any social or supernatural authority (except for retaliatory physical force as a consequence of initiating physical force against others). But what makes consequences "bad"? Except in the Kantian view, there is always an underlying assumption of life as the standard of value (to a significant degree, though not necessarily explicitly so). There is simply no credible way to form such a concept as "bad consequences" except by reference to the requirements of man's life. The choice or desire to live is not arbitrary at all, even in conventional ethical systems. But Objectivism identifies it explicitly and explains why it is so inescapably fundamental. Objectivism also goes on to show how a thriving, happy life relates to "man's life qua man," i.e., how and why positive rewards in life are even more important than the threat of "bad consequences" for bad choices.
Update: Fear and Guilt
A comment asks about fear and guilt in relation to duty. Duty is an idea. Guilt (in this context) and fear are emotions. Emotions proceed from ideas. It is primarily ideas, not emotions, that lie at the base of human motivation. Emotions are just consequences, though they can be reinforcing or paralyzing in the process of acting on an idea.
What, then, motivates people to accept or reject ideas? Why do many people accept the idea of duty? More broadly, why do people accept any code of morality at all? Man has a mental choice about accepting morality of some kind or not; what, if anything, drives him to accept a moral code?
In objective reality, the driving force is the nature of existence and of man, the alternative of life or death confronting man and all other living entities, the need to act in specific ways in order to sustain and strengthen one's life (if one wants to remain alive in the present and throughout the span of one's life), and man's need for the guidance of moral principles in order to identify what to do to live.
Alternative approaches usually reduce to intimidation from others -- seeking to follow whatever they do, and to be accepted by them, and/or surrendering to any demands they may make, in the hope or prayer that they know what they're doing and that it is beneficial for man's life (or perhaps just in the belief that the sheer numbers of human consciouses striving toward common goals will be enough to achieve them).
The first approach, objective reality, is the primacy of existence. The second approach is the primacy of consciousness (usually either entirely social or mixed with a supernatural mythology).
Fear and guilt can arise in either approach, though in different ways. Here is how Leonard Peikoff describes it in The Ominous Parallels, Chapter 16:
Obedience is the precondition of totalitarianism. The preconditions of obedience are fear and guilt; not merely the existential fear created by terroristic policies, but the deeper, metaphysical fear created by inner helplessness, the fear of a living creature deprived of any means to deal with reality; not merely the guilt of committing some specific crime, but the deeper, metaphysical guilt of feeling that one is innately unworthy and immoral.
The role of fear and guilt is also described in Galt's Speech in Atlas Shrugged (pp. 198-199 in the Signet paperback edition of FNI, speaking to a fearful, guilt-ridden audience in a rapidly deteriorating altruist-collectivist society):
It is not any crime you have ever committed that infects your soul with permanent guilt, it is none of your failures, errors or flaws, but the blank-out by which you attempt to evade them.... Fear and guilt are your chronic emotions, they are real and you do deserve them, but they don't come from the superficial reasons you invent to disguise their cause, not from your 'selfishness,' weakness or ignorance, but from a real and basic threat to your existence: fear, because you have abandoned your weapon of survival, guilt, because you know you have done it volitionally.
Galt's audience in Atlas Shrugged isn't necessarily representative of typical American audiences, however. Ayn Rand published an entire article on the American sense of life (versus the European outlook) in 1971, titled, "Don't Let It Go," republished as the concluding chapter in Philosophy: Who Needs It (PWNI, Chapter 18, pp. 279-292 in the Signet paperback edition). The article compares and contrasts the typical American and European sense-of-life, starting with the following (p. 282):
The emotional keynote of most Europeans is the feeling that man belongs to the State, as a property to be used and disposed of, in compliance with his natural, metaphysically determined fate.... He regards service to the State as an ultimate moral sanction, as an honor.... Generations brought up on statist philosophy and acting accordingly, have impanted this in his mind from the earliest, formative years of his childhood.
Regarding the role of guilt, the article explains (p. 289):
The sense-of-life emotion which, in Europe, makes people uncertain, malleable and easy to rule, is unknown in America: fundamental guilt. No one, so far, has been able to infect America with that contemptible feeling (and I doubt that anyone ever will). Americans cannot begin to grasp the kind of corruption implied and demanded by that feeling.
This does not mean that America is safe from transformation into dictatorship over time, however (p. 290):
[A] dictatorship cannot take hold in America today. This country, as yet, cannot be ruled -- but it can explode. It can blow up into the helpless rage and blind violence of a civil war.
Such chaos can lead to dictatorship just as surely as European-style guilt infection, though more indirectly (p. 290), by bringing:
...the longer agony of the collapse of all civilized institutions and the breakup of a nation into roving armed gangs fighting and looting one another, until some one Attila conquers the rest ... chaos as a prelude to tyranny -- as was the case in Western Europe in the Dark Ages, or in the three hundred years preceding the Romanoff dynasty in Russia, or under the war lords regime in China.
Can America still avert such a fate? The article concludes by observing that Americans urgently need to fight for whatever remains of the distinctively American sense of life -- which means understanding its roots in reason and individualism, Enlightenment values inspired most deeply by Aristotle's philosophy (p. 292):
[W]hat we need to fight for is the supremacy of reason, and a view of man as a rational being.
Causality in Morality
In a comment, the questioner asks again about the role of causality in Objectivist morality:
I asked what is the concept that describes why a person should respect the rights of others and has a moral obligation to not do certain things like initiate aggression or steal. Causality obviously does not answer that in the slightest.
The Objectivist view of causality in ethics is indicated in the paragraph of my Answer that begins:
This view of causality in ethics [expressed in the excerpt from Ayn Rand's article] rests on a deeper foundation that includes the nature of the concept "value" and man's life qua man as the only objective standard of value for man. Further elaboration of those points can be found in other Lexicon topics.
If the questioner or anyone else would like a more explicit statement of what the Objectivist ethics says about how man should live, including how he should treat others, the simplest one-word concept would be rationality, which includes justice, which includes the principle of trade as the only rational principle for dealing with others. The basic Objectivist outlook in regard to dealing with others is: "I am rational, because I want to live and reason is my basic means of living; others are rational beings, too, like me; rational beings must deal with each other by rational persuasion and trade, not by physical force, because physical force paralyzes reason and thus is anti-life." Condensed into single concepts, this outlook is integrated within Objectivism's concepts of rationality, justice, trade.
And if anyone then asks what penalty will be imposed on those who initiate the use of physical force against others, Objectivism's answer is: physical force in retaliation against the initiator. But one who seeks to live, and understands that man's life depends on reason, shouldn't need the threat of retaliatory physical force to deter him from initiating physical force against others. He will already understand that others are rational beings, too, like himself, and that initiation of physical force against rational beings (whether himself or others) endangers or destroys their lives. He will understand that he can gain enormous benefits for himself by trading with others, and that initiating physical force against them is counter-productive; they will not want to provide anything of value to him if he attacks them, and they will want instead to retaliate against him to exact restitution if possible, and retribution (i.e., justice)). Again, the most immediately explicit concept to follow is trade (founded on reason), not force.
Causality, in turn, means that one chooses all of one's goals, including one's relations with others, on the basis of the virtues and values identified by Objectivism, which proceed from man's life qua man as the only objective standard of value for man.