This request hails from the early days of Stack Overflow, and was one of the first and most frequent feature requests to be declined.
The official reference thread is A "friends list" on Stack Overflow would be niceA "friends list" on Stack Overflow would be nice on Meta Stack Overflow.
What it essentially comes down to is what you've said - Stack Exchange isn't a social networking site. We don't want to be. The focus has always been on the content of the questions and the answers, not who's posting them. Following individuals would go against that concept. We don't want to encourage cult followings, we want to encourage high-quality content regardless of who wrote it.
You can subscribe to a specific RSS feed for each user, so if you really want to follow a user's contributions, you could do it that way. The format is https://superuser.com/feeds/user/userID
. Yours is https://superuser.com/feeds/user/54785, for example.