Depends what country you're asking about. Different countries have different laws relating to (a) inheritance of noble titles and (b) the rights of illegitimate children in general. In Spain, for example, when a titled person dies the title does not pass automatically to *anyone*; anyone who considers themself the next heir petitions the monarch to re-grant it to them, and if there are competing claims it's up to the monarch to decide. And in France *all* a person's children have equal property inheritance rights, so if a man has an illegitimate child, even one he kept secret from the rest of his family or didn't even know about, after his death that child can claim an equal share of his estate, if they can prove their parentage.
However, in Britain, under the common law an illegitimate child was traditionally considered "filius nullius", which is Latin for "nobody's child", and had no right to any inheritance from *either* of its parents. That meant that if a man wrote in his will that he was leaving property to "my children", that meant only his legitimate children by his lawfully married wife. This harsh law wasn't changed till 1969. Now "my children" is held to include illegitimate children also (unless specifically stated otherwise), and it is understood that illegitimate children have a reasonable claim and expectation of property inheritance. But illegitimate children, and adopted children' are still absolutely excluded from inheriting any title of nobility.
When a British peer dies without a legitimate son to inherit his title, what happens next depends on the terms under which that specific title was granted. (For centuries, titles were granted by the monarch in a document called a 'Letter Patent' which set out the terms of the grant including exactly how it was to be inherited.)
- The vast majority of letters patent simply say 'to his heirs male of the blood', which means that every male descended in the legitimate male bloodline of the original grantee is in the line of succession. So if the current holder dies without a legitimate son, the title goes to his younger brother (or, if that brother is already dead, that brother's son). If there is no younger brother, you go one generation further up to his paternal uncle (or paternal uncle's son/grandson). And so on: the title only dies out when there is nobody at all in the legit male bloodline of the original grantee. But women, or anyone descended through the female line, are specifically excluded.
- If the original grantee had no legitimate sons and didn't look likely to have any, sometimes the letter patent included what is called a "special remainder"; a clause saying that after his death the title could be inherited by his eldest daughter or even (as in the case of Admiral Nelson, who was estranged from his lawful wife and living in sin with Lady Hamilton) his brother. But special remainders are only one-offs; after that first occasion the title obeys normal rules.
- Some British peerages, mostly very old ones, simply go to "heirs of the blood", which means either male or female heirs. But daughters only inherit if there are no sons. And if there is more than one daughter, the rules in Scotland differ from England-and-Wales:
- In Scotland, the eldest daughter gets the title (and any family estate that goes with it) and can pass it to her heirs. Simple.
- But English nobiliary law does not recognise seniority among daughters; if a man has only daughters, they all have an equal claim to share the inheritance. However, two or more people can't share a noble title. What happens in this case is that the title "goes into abeyance" - essentially, it is frozen until all but one of the daughters' bloodlines have died out, leaving only one claimant. This can take a very long time: one or two medieval peerages only came out of abeyance in the 20th century. (This is why it's mostly only Scottish peerages that allow inheritance by daughters; English peerages are generally restricted to "heirs male" simply because abeyance is such a headache.)