HackerOne Reputation – Increasing Valid Bug Submissions and Signals

Edited on 12/11/2015 to reflect the latest Reputation implementation.

One of the primary challenges when running a vulnerability coordination program is distinguishing signal from noise. Our former colleagues at Facebook evaluate over 20 invalid submissions for each valid report – that’s only 4.6% signal! The programs hosted at HackerOne have fared a bit better: on average 19% of reports are valid, but some outliers deal with as low as 6%. This noise is undesirable for everyone, driving up response time, introducing unnecessary latency in resolving security issues, and increasing the likelihood that valuable signal will get lost.

The HackerOne vulnerability coordination platform provides security teams with the tools to overcome these challenges. Today, we’re turning a beta feature live for everyone: a new reputation system that makes running a program even easier. This system gives additional recognition to the best researchers while more quickly surfacing quality reports to security teams.

The vast majority of security researchers generate reports of consistently high quality. Our analysis shows that the remaining noise stems from a classic tragedy of the commons: a minority of researchers with low confidence submissions hoping to stumble upon a success. But many of these researchers learn and improve with each failed attempt. The challenge for security teams is that past performance is not indicative of future results.

This new reputation system will help provide security teams with the means to more effectively act upon the invaluable information shared by the security community. As a researcher submitting vulnerabilities through the HackerOne platform, your reputation measures how likely your finding is to be immediately relevant and actionable. Reputation is gained or lost based exclusively upon your track record as a researcher.

How it works

You gain reputation when:

• Your report is Closed as Resolved: +7
• Your report is Closed as Duplicate (Resolved): +2. Only applied if reported before the original was closed.
• You are awarded a bounty. The amount is based on standard deviation from the program’s mean:
  • +50: $ >= µ + 1σ
  • +25: $ > µ
  • +15: $ >= µ – 1σ
  • +10: $
• Your report is Closed as Informative: 0
• Your report is Closed as Duplicate (Informative): 0

You lose reputation when:

• Your report is Closed as Not Applicable: -5
• Your report is Closed as Duplicate (Not Applicable): -5
• Your report is Closed as Duplicate (Resolved and Public at time of submission): -5

Other details:

• We believe everyone deserves the benefit of the doubt. Researchers begin with a reputation of 100. Reputation cannot decrease below 0.
• You’ll always have access to a detailed log of reputation history.
• Reputation will never be necessary to access core functionality on the platform to ensure it remains accessible to new or anonymous users.
• Our approach draws inspiration from two communities we’re fond of.

The most visible manifestation of the new reputation system will be its usage as a ranking mechanism for the many Thanks pages on the platform. In the near future, we’ll also be announcing a number of privileges that are gained by maintaining a high reputation, such as becoming eligible to receive invitations to private bounty programs.

Finally, and most importantly, we’re now tying our rate limiting system directly to reputation. Should your reputation decrease, the system will gradually reduce the number of submissions allowed in a given time period. We believe it is critical to this community that response teams be afforded a high-signal environment so that they can focus on providing a quality response to researchers who turn in the best vulnerabilities.

HackerOne is committed to empowering the world to build a safer Internet, and building the most useful platform for vulnerability coordination is central to that mission. We welcome feedback on this new reputation system, and we hope response teams and researchers will enjoy the benefits of even higher quality reports and faster response times.

HackerOne is the #1 hacker-powered security platform, helping organizations find and fix critical vulnerabilities before they can be criminally exploited. As the contemporary alternative to traditional penetration testing, our bug bounty program solutions encompass vulnerability assessment, crowdsourced testing and responsible disclosure management. Discover more about our security testing solutions or Contact Us today.