Threshy: Supporting Safe Usage of Intelligent Web Services
Increased popularity of `intelligent' web services provides end-users with machine-learnt functionality at little effort to developers. However, these services require a decision threshold to be set which is dependent on problem-specific data. Developers lack a systematic approach for evaluatin...
Saved in:
Main Authors | , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
19.08.2020
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Increased popularity of `intelligent' web services provides end-users with
machine-learnt functionality at little effort to developers. However, these
services require a decision threshold to be set which is dependent on
problem-specific data. Developers lack a systematic approach for evaluating
intelligent services and existing evaluation tools are predominantly targeted
at data scientists for pre-development evaluation. This paper presents a
workflow and supporting tool, Threshy, to help software developers select a
decision threshold suited to their problem domain. Unlike existing tools,
Threshy is designed to operate in multiple workflows including pre-development,
pre-release, and support. Threshy is designed for tuning the confidence scores
returned by intelligent web services and does not deal with hyper-parameter
optimisation used in ML models. Additionally, it considers the financial
impacts of false positives. Threshold configuration files exported by Threshy
can be integrated into client applications and monitoring infrastructure. Demo:
https://bit.ly/2YKeYhE. |
---|---|
DOI: | 10.48550/arxiv.2008.08252 |