Here are two ways to think about mathematics (among possibly others). (1) As a discipline: an institution with some degree of political organization, a history, etc. (2) As an intellectual virtue: a habit of the cognitive faculty which consists in the enhanced potential to carry out certain cognitive acts. (I say enhanced potential because a faculty in itself is already a potential.)
It is in sense (2) that mathematics can be said to have objects. A cognitive act in general is a representation, and a representation, if at all successful (if not, e.g., self-contradictory) has some at least possible being as its object. If there is such a thing as mathematics at all in sense (2), then there must be something which characterizes the objects of just those acts for which it is the enhanced potential.
(Note Im using object in the correct, relative sense: an object is the object of something. According to idealists of various stripes, every being, or at least every contingent being, must be the potential object of some cognitive faculty. But for just that reason its a bad idea, especially if, like me, youre an idealist of some stripe or other, to use object as a synonym for being.)
It would be wrong to define the intellectual virtue (2) solely in terms of the institution (1). We should want to say that every normal human, past some fairly young age, possesses (knows) at least some mathematics. On the other hand, the people who play the leading role in institution (1) that is, professional mathematicians owe their position, on the whole, to being very good at something. There may be some disciplines where success depends, not on any cognitive ability, but rather, for example, on political or social skills. But if you know any even moderately good mathematicians you will know for sure that their skills dont mostly lie in that direction. We should want the intellectual virtue (2), which we all share to some extent, to be the one successful mathematicians must have to an unusual extent.
But this puts constraints on the objects of mathematics. For example: although some may think that mathematicians are particularly good at counting to very high numbers, that is not actually the case. Hence, the cognitive act of representing a given (high) finite cardinality is not as such an act of the virtue (2). It follows that a given natural number simply as such (unless perhaps it is very small) is not an object of mathematics.