The new STEMsocial engagement/reputation system
For those who are following the STEMsocial distilled series, you should have now noticed that our weekly posts now contain two top-five lists of some special community members we want to highlight each week.
The first of these two lists includes our five most engaging community members. The second one includes the five authors generating the largest amount of interactions. All mentioned authors are shown together with their respective engagement and interaction scores.
With the present post, I present how those engagement/interaction scores are computed. For now, the whole method is quite experimental, and I will emphasise this several times in the post.
However, the idea behind this article is to share information in order to get feedback to improve the way the computation of the scores is made. I will carefully read and address anything raised into the comments to this post, and implement/test the proposals out.
The algorithm
All posts that qualify to the computation of the engagement/interaction scores are those posts supported by STEMsocial over the last 4 weeks, regardless the weight of the STEMsocial support. Then, any comment to those posts supported by STEMsocial is accounted for to generate engagement/interaction points.
However, not all comments generate points. We begin with getting rid of the not so useful comments (like ‘thanks for your post’). To this aim, one filters away all comments featuring less than 20 words or 125 characters. Moreover, a few accounts are excluded, essentially bot and notification accounts.
Next, the number of points assessed to any of the remaining comments consists of the square root of the number of characters making the comment. This is probably the weak spot of the whole process, but one must start somewhere. Feel free to suggest anything better.
For a given comment, the number of points is given both to the comment author as engagement points, as well as to the author of the initial post as interaction points. In this way, each STEMsocial member who contributes to the interactions within our community gets points: the author and the engaged members.
For what concerns the authors, the interaction scores are further averaged over the number of posted articles.
The results
| 1 | @lemouth | 1842.339 |
| 2 | @amestyj | 994.366 |
| 3 | @agmoore | 816.442 |
| 4 | @gentleshaid | 642.135 |
| 5 | @lupafilotaxia | 570.482 |
| 6 | @acont | 483.039 |
| 7 | @nesomi | 310.746 |
| 8 | @eliaschess333 | 277.132 |
| 9 | @rbalzan79 | 276.037 |
| 10 | @carlos84 | 262.911 |
| 11 | @tsoldovieri | 258.015 |
| 12 | @scholaris | 257.085 |
| 13 | @capp | 241.253 |
| 14 | @chappertron | 237.314 |
| 15 | @emiliomoron | 190.295 |
| 16 | @robotics101 | 183.023 |
| 17 | @wilmer14molina | 160.738 |
| 18 | @steepup | 158.512 |
| 19 | @abneagro | 155.658 |
| 20 | @mike961 | 149.291 |```
In addition, the authors generating the largest amount of interactions are, given together with their scores:
```| # | Member | Score |
| 1 | @lemouth | 648.309 |
| 2 | @scholaris | 300.231 |
| 3 | @chappertron | 296.442 |
| 4 | @lemouth-dev | 257.277 |
| 5 | @whack.science | 256.019 |
| 6 | @abneagro | 236.182 |
| 7 | @agmoore | 235.816 |
| 8 | @artemislives | 173.656 |
| 9 | @gentleshaid | 168.031 |
| 10 | @enforcer48 | 167.438 |
| 11 | @amestyj | 163.692 |
| 12 | @faltermann | 153.294 |
| 13 | @anaestrada12 | 135.761 |
| 14 | @steemstem | 110.988 |
| 15 | @aleestra | 107.595 |
| 16 | @pearlumie | 102.662 |
| 17 | @menoski | 91.550 |
| 18 | @tsoldovieri | 89.860 |
| 19 | @mike961 | 79.739 |
| 20 | @mengene | 78.742 |```
<hr />
I am definitely not happy with the way our engagement indicators are computed. However, as I said above, we need to start somewhere. I will now listening about how to improve this. Please feel free to speak up (or write something in the comments to this post)!
If you like what we do, please consider:
- [supporting our funding proposal](https://hivesigner.com/sign/update-proposal-votes?proposal_ids=%5B91%5D&approve=true") (#91), on which extra information can be found [here](/@lemouth-dev/rebooting-and-clarifying-the-stemsocial-proposal);
- [approving](https://hivesigner.com/sign/account_witness_vote?approve=1&witness=stem.witness") our witness (@stem.witness);
- delegating to the @stemsocial account (or trailing it).
</div>
Err umm...thank you for your post!
The beginning is usually the most difficult part of the process. Yet, a consistent and reliable standard of analysis is required for any process. It’s good for users to recognize how the system begins to review their work. While written by humans, the program performs its actions only as directed and to the extent reported by your article.
It’s a great step. Thank you for creating it.
Thanks for your comment! We are still unsure about what to do with those numbers. So if you have ideas/suggestions, feel free top share them!
Note also that this does not enter in the post reviewing by our curators.
I think it's a good enough start. I gather that the goal is to measure and judge with as little human opinion as possible. I do similar evaluations at work. Though I feel it is impossible to rule out errors completely.
Tagging @abh12345 to follow-up a discussion started weeks ago ;)
Just finished reading on the stemgeeks site :)
I think this is a good starting point. Including only the posts that have earned support from the community account is a good filter.
A while back, I toyed with discounting comments shorter than 20 or so characters but decided against it in the end. Instead I occasionally check for people repeating the same short (or longer) phrase, and actually, there is very little of that.
Good luck :)
I tried without the filter, and indeed we see a few permutations in the ranking, but no large change (in particular as we have the blacklist with the bots).
I am still thinking about what to do with the sqrt metric, that is possibly not ideal. But for now, I have no clue.
PS: the post can also be read on the STEMsocial app. ;)
I think you'll have to try it and see, there's no harm in adjusting metrics in the future.
I was 25th I guess this time. +_+ this is gamification now.
I have got some ideas btw for generating activity in the chain from our discord server itself. Maybe we should think about that too. One idea is a bot like gina where people can interact with it via replies. Another one is perhaps a science trivia and puzzle activities that can be broadcasted on to the chain.(without spamming)
I think lot of us are hooked to discord. So thought why don't we give focus to that and link it to the chain.
You are getting higher and higher in the rankings... ^^
For the rest, those could be nice ideas, but we must be careful with spamming. Also, the activities should be appropriate for the on-chain format, and we may need to think about the reasons behind having that on chain and not off chain. Anyways, this is good food for thoughts! :)
Interesting analysis!
I'm Nr. 3 in interactions !?!
Pretty nice ;-)
Yep. Apparently your posts generate a lot of comments! ^^
I believe as times goes on, it will be much more improved...
For now, I can't say much. I just feel this will trigger and encourage more members to be active..
A good way to start
I hope this will bring some motivation for people to interact with each other even more. Let's see where this will lead us :)
And if you have idea on a later stage, I am here to listen. Always! :)
I'm not so at peace with the square root of number of characters thing. A short comment can actually weigh more than a long comment filled with gibberish. We need both quantity and quality and the 125 characters/20 words filter seems enough imo. Further finding the square root of number of characters might encourage spamming/plagiarism and how many people have got the time to check comments for plagiarism?
Interesting thoughts! Really!
I am not sure people will really try to spam/plag to increase their score as this will bring them nothing, at least for now. Moreover, people finishing high on the list will be automatically scrutinised, and blacklisted from the script if needed (so that it may work... maybe once :D ). I don't know here... Let's continue discussing this.
Back to the square root now. I chose this as this decrease the relative importance of very long comments. Moreover, with the 125-characters threshold, the smallest score a comment could bring is 3 points so that too short comment are not really impacted as they don't count.
First, as suggested by @abh12345 below, I have tried to extract the scores without the word count / length limit. Not much changes in terms of the top-20. We have some permutations, but actually nothing more. I may actually remove this filter. I am still thinking about it.
Second, I agree the sqrt may not be ideal. @borislavzlatanov proposed to include the number of votes a comment brings. I however don't know what to do in practice. Any idea for another metric?
Maybe include the number of votes a given comment has? This can further boost engagement and comment curation (which I think is largely missing but super important).
I agree comment curation is missing. I usually cast a few votes on comments to posts I read. And @steemstem upvotes most comments posted through our app (i.e. not many of them, so far ;) ).
So to come back to your suggestion, you suggest to add some bonus points to any comment triggering one or more votes? If yes, how to assess those bonus points? Any idea? A global factor of (1 + n_votes/10) multiplying the comment score?
Well, I think the question is what behavior is desired. What would you like to see happen? For example, if you value people posting quality comments and keeping doing it, then the metric may be, for example, the average number of upvotes on the person's comments in the StemSocial community during, say, the last 3 months. Or something like that. The metric set this way would mean that those people who have recently made quality comments will have a higher reputation. But also, they will lose that reputation if they don't keep it up, or maybe new members can easily come and get a high reputation if they engage with high-quality comments.
Now, the next question will probably be: how much importance do you place on comment quality? If high importance, then it might have let's say 50% weight in the reputation algorithm. Or it could be 20% if it's only one of five things you value.
I just came up with this on the spot, surely it can be improved. But I think starting this way, with such questions, will probably be better than starting with the precise math algorithm. Once there is clarity on what is valued, and how much, then the precise math can follow from that.
The problem is well known here: we cannot connect quality and upvotes, especially for comments. For that reason, we should be careful.
Definitely. Implementation takes 5 seconds. Defining what to implement: days or weeks :)
Yes, for sure, upvotes on a comment doesn't mean comment quality. Further brainstorming about what things will more reliably indicate comment quality... maybe an upvote from the post author. But what if the post author upvotes comments out of politeness? That might happen less with authors having more education about the algorithm. This might work, although it certainly seems like a weak spot.
But what if both an upvote from the post author and upvotes from others are considered in a combined fashion? So that only one of the two wouldn't hold much weight, but if both are present, then it holds larger weight? Maybe this will work more reliably. And also upvote weight can be considered - is it a 100% upvote or a 5% upvote? Getting upvotes with a high weight from both the post author and others seems more reliable to me. What do you think?
How can one not engage in a post on engagement :))
I think profiling engagement is a good idea. As an author, I know comments are almost more appreciated than votes. We work so hard on our posts, invest so much of ourselves, it's heartening to know someone is affected by our efforts.
This system is a reminder to all of us that engagement matters.
Thanks for engaging on this engagement post ;)
I am personally trying to comment on every single post I read. Usually, I always have something to say or to ask. Even if it is trivial. As an author, I love comments. This lies among the most important reasons for blogging, in my opinion. I guess it is the same for most other authors.
Bearing this in mind, the rest (i.e. my commenting behaviour) follows ;)
Thanks for your contribution to the STEMsocial community. Feel free to join us on discord to get to know the rest of us!
Please consider supporting our funding proposal, approving our witness (@stem.witness) or delegating to the @stemsocial account (for some ROI).
Thanks for using the STEMsocial app, which gives you stronger support. Including @stemsocial as a beneficiary could yield even more support next time.
I am relatively new to the community. And i must admit, the guys in charge of the community are doing quite an absolutely amazing job! I believe the engagement system will definitely work well and i commend the initiative @lemouth-dev. Keep it up boss!
Thanks for your nice words, and again welcome here! Little by little we will build things up!
:)
Greetings to all colleagues in the community, I think that valuing a publication by the comments received is fundamental, as it is an indicator of how interesting the content published, particularly I find encouraging when receiving comments in publications, that motivates the researcher, I think they are addressing a key point inside hive, because a social network is to interact and leave insights on content that we find interesting. It seems to me that any measure or methodology they adopt is a big step that will benefit the community and the filters will undoubtedly generate results on the behavior of the users on the platform !
I agree: comments are what motivates me! I also try to upvote a little the comments I see that are interesting, as a kind of symbolic sign of recognition to the author.
Interestingly enough, the filters have very little impact (see my answer to some of the other comments here :) ). There are a few swaps in the rankings, but not much on top of this.
See the maths part of human interaction stumps me. People say I should do something like it for @naturalmedicine but I'm not a dev.. this is impressive. Still, there's the heart and there's numbers. Well done for trying to marry the two.
I find it is always cool. A little bit of gamification in this online world ;)
Note that if you provide me the exact rules you want to implement, I would be happy to code/run a script for @naturalmedicine.
That'd be really amazing @lemouth - we will definitely ask you and it means a lot that you offered!
I am completely sure that from this the comments in the publications will increase considerably, I think that people should feel motivated with this initiative.
I have a question Ben, does the algorithm filter comments in all languages? I know that in Spanish yes because I am looking at several authors from our Hispanic community in the list. But German, Italian, Chinese, French, etc etc ?
I hope people will get motivated as well. We however still need to brainstorm about potential rewards (even symbolic ones). Any suggestion is welcome.
There is no language filter here. All languages are competing with each other. Chapper is there for instance, and I think he comments mostly in German. hat you see is probably related to the size of the various sub-communities. The French sub-community is barely existent, so that this is not surprising no French-speaking author are part of the top 20.
Great then, I think I told you some time ago that we reward weekly the best 5 authors with the most positive comments (related to the topic of the publication) this could work and with this new system everything is easier because it is automatized!
Wouldn't this require comment curation (to select the best comments), and therefore more work for our curators? When we are dealing with rewards, we must be 10 times more careful and we should think twice before automating the process, shouldn't we?
In that case I think you are right, I read the comments of all the publications and set a score on a 1 to 5 pts basis depending on the type of comment. Maybe the curators if we agree we can divide the work by area, like for example: Shaid and Chapp the publications of biology, health, Matowl mathematics, I can take care of the ones of physics, chemistry, engineering and so on. It all depends on the availability of each curator. It's just a proposal, it doesn't mean it's like that, there may be other options
This seems to me an excellent initiative, I came back a few months ago after almost a year without posting and I admit that I have noticed a decrease in the amount of comments received, and interactions in general, although I am happy to see that many users I read are still active.
It is true that the system may require some adjustments, but it seems like a good start! There is nothing like a comment that makes you feel that someone genuinely enjoyed your content.
I have picked your post for my daily hive voting initiative, Keep it up and Hive On!!