Expanding Reputation: Introducing Signal and Impact

Edited on 4/8/2016 to reflect the latest Signal and Impact implementations.

We first introduced Reputation in October 2014 to provide additional recognition to the best researchers, and to highlight quality reports to security teams. Each researcher has a Reputation based on their track record on HackerOne. Reputation has been continuously improved since its introduction and has become integral to the platform.

To build upon its usefulness as a ranking system, we launched two new dimensions to better show how each researcher’s Reputation has been achieved: Signal and Impact.

Why We’re Improving Reputation

The existing Reputation system is effective at measuring a consistent track record of valid results, serving as an accurate ranking function. However, it occasionally obscured quality over quantity. The introduction of Signal and Impact helps further highlight quality performance.

New Dimensions: Signal and Impact

Signal is the average Reputation per report.
Reputation is gained or lost each time a report is closed, making Signal an aggregate representation of report validity.

Impact is the average Reputation per bounty.
Reputation is gained based on the relative size of the awarded bounty, making Impact an aggregate representation of report severity.

• Signal and Impact are only calculated when there are more than 3 closed reports or 3 bounties, respectively.
• Signal is measured on a scale from -10 to 7, corresponding to Reputation changes for triage states ranging from “Spam” to “Resolved.” Self closed reports and Duplicate reports are excluded.
• Impact is measured on a scale from 0 to 50, corresponding to Reputation gains calculated by bounty levels awarded.
• Impact is only calculated for reports submitted to programs that offer bounties. We also display the percentile rank for Signal and Impact, relative to other researchers on the platform.

Examples

Alice and Bob have the exact same amount of Reputation (241). But Alice has a higher Signal (4.12 > 1.35), indicating a better track record of submitting valid reports. Alice also has a higher Impact (15.83 > 9.50) and therefore, a higher chance of submitting an important report.

This is valuable information for both of them. Alice will see that she is doing quite well with her current trajectory, while Bob will see that others are more accurate about their reports, and can adjust his reporting processes accordingly.

Conclusions

For a response team, Signal provides a means of identifying researchers who have had consistently valid reports while Impact highlights those with the greatest severity. For researchers, we hope both Signal and Impact will help you benefit through an improved understanding of your performance relative to your peers. In the coming months, we’ll be incorporating Signal and Impact into other visible areas of HackerOne, such as the invitation system. We hope you’ll find them useful and as always, we’d love to hear your thoughts.

HackerOne is the #1 hacker-powered security platform, helping organizations find and fix critical vulnerabilities before they can be criminally exploited. As the contemporary alternative to traditional penetration testing, our bug bounty program solutions encompass vulnerability assessment, crowdsourced testing and responsible disclosure management. Discover more about our security testing solutions or Contact Us today.