Comment on page
Definitions
Software that generates the Smart Bounty (see definition below) for a given job. Responsible for launching new bounties into the blockchain for the Job Requester, sending manifest to Reputation Agent for signing, encrypting signed manifest. MVP: hosted by us behind REST API. Soon: open source.
Website that is serving task to its users, embedding a script hosted on an Exchange. https://exchange/mine?difficulty=3&safe_content=1
User of website who answers tasks embedded on website of mining server.
Components:
• REST API gateway.
• Blockchain monitor.
• Job allocator.
Serves up a script with task functionality. (Generates JS with correct embedded vars from job.)
Keeps recent mining server stats, mining client request counts for graylisting, whitelist/blacklist; caches reputation scores computed by recording oracle and uses to stop bad actors early. Watches blockchain for new jobs.
Chooses job from open bounties based on mining server criteria.
Fulfils mining client request with job data: URL to image, question, recording oracle server URL Maintains sets of clients fulfilling Restricted Audience criteria, serves clients tests if necessary If no jobs available: acts as captcha server using fallback
Receives compensation based on requests served via smart bounty. Publishes current bid prices for each task type, denominated in HMTs.
REST API gateway.
Receives mining client response, records response, returns yes/no (did the mining client succeed in their tasks?), weight, reputation update.
Has access to job data.
Each question gets asked N times. Pass weight starts at 0, then performs rolling computation of inter-rater agreement, confidence based on client + server reputation for agreement once N>2. Stores final mining server result contributions in immutable form.
Updates blockchain with URI, hash once all results recorded to specified N per question. Receives compensation based on requests served via smart bounty.
Optional: needed only to enable seamless onboarding for Exchange users with no native ETH addr. Creates ETH public/private keypair, returns public key address to Exchange to use as ‘secret’ in the reCAPTCHA lingo.
The requester of a human evaluation job, i.e. the machine learning model builder.
A smart bounty is a service contract initiated by the requester via the Contract Originator component that holds HMT in escrow. Contains:
1. Type of request: image labeling, etc
2. Number of answers sought per dataset point
3. Location of dataset manifest (URLs of images, question(s), etc) - location of validation set manifest (URLs with answers)
4. Readable only by reputation oracle - bid amount
5. Expiration date
6. Valid exchanges for request
7. Valid reputation agent for request (exchanges may have their own whitelist)
8. Valid recording oracle for request (exchanges may have their own whitelist)
9. Valid reputation oracle for request (exchanges may have their own whitelist) - whether content is “safe” for all ages (ignored until requester rep is >N)
10. Optional: valid mining servers, restricted audience
11. Optional: minimum mining server trust score
Below is a high level sketch of functionality. Please refer to the hmt-contracts repo’s smart bounty example code and library for current implementation details.
Function: jobIsValid(bool) callable only by specified reputation agent.
• Confirms that the job should be processed by an exchange: exchanges maintain their own whitelists of approved reputation agents, which can be derived from on-chain reputation scores.
• If false, bounty is canceled and requester receives balance in escrow, less fee.
Function: storeIntermediateResults(URI or IPFS-style key, hash) callable only by specified recording oracle.
• Recording oracle encrypts results with reputation oracle’s public key.
Function: storeFinalResults(URI or IPFS-style key, hash) callable only by specified reputation oracle.
• Reputation oracle encrypts results with requester’s public key.
Function: jobSatisfied(bool) callable only by specified reputation oracle.
• Triggers release of results to requester, payment of oracles and exchange.
Function: distributeBounty(miningserverAddress, amount) callable only by specified reputation oracle.
• Distributes specified amount of tokens from escrow to mining server each time it’s called.
May be signed by reputation agent to prove some sort of validation has been performed on the request.
%204.00.00%20p.%20m..png?alt=media&token=66c39657-0e77-432e-b18b-05f7dce7f5c2)
1. Requester creates smart bounty (service contract) via Contract Originator with HMTs in escrow.
• Manifest optionally signed by Reputation Agent, vouching for requester without reputation; Contract Originator sends manifest to Reputation Agent via REST API.
• Reputation Agent performs verification of manifest contents, e.g. runs images through adult content detection.
2. Exchange watches blockchain, picks up job.
3. Mining server embeds script from exchange.
4. Mining client:
• Hits task request on mining server (website)
• Gets dataset entry from location specified by exchange, i.e. image URL - Answers question, standard reCAPTCHA protocol follows:
• Client hits POST on request, passes ‘g-recaptcha-response’ to mining server.
• Mining server sends request to recording oracle - ‘secret’ (ethereum key in our case)
• ‘response’ (from mining client) • Mining server gets go/no go from recording oracle (‘success’ field in response)
• Recording oracle stores results of mining server contributions per job in immutable form.
• Reputation oracle evaluates results, makes determination about whether to release funds.
5. Reputation oracle calls smart bounty function jobSatisfied(true) to release results to requester, pay oracles and exchange.
6. Reputation oracle calls smart bounty function distributeBounty(addr, amount) for each mining server to send tokens from escrow to mining server addresses.
7. Reputation oracle handles earned amounts too small to economically transfer:
• MVP: Reputation oracle sweeps remaining aggregate of earned amounts too small to pay for individual transfers into offline holding account, writes out ownership data for periodic review and distribution once total earned by a mining server across jobs is above threshold.
• Later: maybe Plasma when it’s ready.
(online/streaming mode)
1. Requester calls Exchange REST API reserveAPIKey , gets API key to include in manifest.
2. Requester creates smart bounty (service contract) via Contract Originator with HMTs in escrow, effectively putting a deposit down on future API requests.
• Manifest includes reserved API key.
• Manifest includes webhook endpoint controlled by Requester to receive streaming results.
• Manifest optionally signed by Reputation Agent, vouching for requester without reputation; Contract Originator sends manifest to Reputation Agent via REST API.
• Reputation Agent performs verification of manifest contents, e.g. runs images through adult content detection.
3. Exchange watches blockchain, picks up job. - API key is now active.
• Requester uses API key to submit tasks directly to Exchange via REST API.
4. Exchange may perform additional verification of task contents, e.g. running images through adult content detection.
• Exchange schedules tasks directly at current bid price for the task type to ensure lowest latency. • Exchange tracks total spend, expires API key when funds are depleted.
• Mining server embeds script from exchange.
5. Mining client:
6. Hits captcha request on mining server (website)
• Gets dataset entry from location specified by exchange, i.e. image URL.
• Answers question, standard reCAPTCHA protocol follows:
• Client hits POST on request, passes ‘g-recaptcha-response’ to mining server - Mining server sends request to recording oracle
• ‘secret’ (ethereum key in our case) • ‘response’ (from mining client)
• Mining server gets go/no go from recording oracle (‘success’ field in response)
• Recording oracle stores results of mining server contributions per job in immutable form.
• Recording oracle immediately forwards each response to Requester along with confidence score.
7. Reputation oracle evaluates final results, makes determination about allocation of funds.
8. Reputation oracle calls smart bounty function jobSatisfied(true) to release aggregate results to requester, pay oracles and exchange.
9. Reputation oracle calls smart bounty function distributeBounty(addr, amount) for each mining server to send tokens from escrow to mining server addresses.
10. Reputation oracle handles earned amounts too small to economically transfer:
• MVP: Reputation oracle sweeps remaining aggregate of earned amounts too small to pay for individual transfers into offline holding account, writes out ownership data for periodic review and distribution once total earned by a mining server across jobs is above threshold.
• Later: maybe Plasma when it’s ready.
(Legacy Systems)
1. Create dataset of possible images, texts, etc.
2. Manually label some number of those images, reserve as validation set.
3. Upload to MT etc for labeling, object masking, etc.
4. Get label results.
5. Compare results to validation set to produce accuracy score, run again if below threshold.
(batch)
1. Create dataset of possible images, texts, etc.
2. Manually label some number of those images, reserve as validation set.
3. Create job:
• Choose a minimum accuracy score (enforced via validation set and inter-rater agreement)
• Upload dataset and validation set
• Place smart bounty at bid informed by current jobtype bid on chosen Exchange
4. Get label results from recording oracle or webhook delivery.
(online/streaming)
1. Reserve API key from Exchange.
2. Create job:
• Place smart bounty to spend up to N HMTs on Exchange.
• Specify webhook to receive streaming results.
3. Send API request to Exchange with individual task(s), datapoint(s).
4. Get label results from webhook delivery immediately.
Last modified 10mo ago