Roboflow
ProductForumComputer Vision DatasetsComputer Vision Models
Search
⌃K
Links

Web Browser (On Device)

Realtime predictions at the edge with roboflow.js
For most business applications, the Hosted API is suitable. But for many consumer applications and some enterprise use-cases, having a server-hosted model is not workable (for example, if your users are bandwidth constrained or need lower latency than you can achieve using a remote API).
That's where roboflow.js comes in. We have built a custom layer on top of Tensorflow.js to enable realtime inference via JavaScript using the same state of the art models from Roboflow Train that you're used to.

Learning Resources

Video Demo

Once you've trained a model, you can try it using your webcam using the "Try with Webcam" link. This sample app is available for you to download and tinker with the "Get Code" link. A simple hand-detector webcam demo is available here (it is trained on the public EgoHands dataset).
Demo video of using roboflow.js to build a custom web app.

Interactive Repl.it Environment

We have published a "Getting Started" project on Repl.it with an accompanying tutorial showing how to deploy YOLOv8 models using our Repl.it template.

GitHub Template

The Roboflow homepage uses roboflow.js to power the COCO inference widget. The source code for this project is available on GitHub. The README contains instructions on how to use the repository template to deploy a model to the web using GitHub Pages.

Installation

To add roboflow.js to your project, simply add the script tag referencing our CDN to your page's <head> tag.
<script src="https://cdn.roboflow.com/0.2.20/roboflow.js"></script>

Initialization

To authenticate with our API and load your model you will need your publishable_key, and your model's ID and version number. The simplest way to obtain these values is by clicking Try with Webcam after using Roboflow Train and clicking the "Get Code" button from the demo site; the snippet is pre-filled with the necessary values.
Alternatively, you can obtain your publishable_key from the Roboflow API settings page. Your model's ID and version number are located in the URL of the dataset version page (where you started training and see your results).
Note: your publishable_key is used with roboflow.js, not your API key (which should remain secret).
The roboflow.auth().load() function returns a Promise that you can use to access a loaded model object. If you plan to use your model in one place on the page, you can add a .then() statement directly after the .load() function like this:
roboflow.auth({
publishable_key: "<< YOUR PUBLISHABLE KEY >>"
}).load({
model: "<< YOUR MODEL ID >>",
version: 1 // <--- YOUR VERSION NUMBER
}).then(function(model) {
// model has loaded!
});
You can also define an async function that returns the model on which you can run inference:
async function getModel() {
var model = await roboflow
.auth({
publishable_key: API_KEY,
})
.load({
model: MODEL_NAME,
version: MODEL_VERSION,
});
​
return model;
}
​
var initialized_model = getModel();
​
initialized_model.then(function (model) {
/// use model.detect() to make a prediction (see "Getting Predictions" below)
});
This code lets you initialize your model in one place for use throughout your code.

Configuration

The model object has a configuration method you can use to tune its predictions. It takes an options object with three (optional) keys:
  • threshold - (float; 0-1) the minimum confidence required to return a box. The higher you set this value, the less false positives you'll receive (but also the more false negatives).
  • overlap - (float; 0-1) the maximum area two boxes of the same class are allowed to overlap before the least confident one is recognized as a duplicate and dropped.
  • max_objects - (int, 1-Infinity) the maximum number of boxes to return.
model.configure({
threshold: 0.5,
overlap: 0.5,
max_objects: 20
});

Getting Predictions

To get predictions back from your model object, use the detect method. It takes an input image (which can be any <img>, <canvas>, or <video> element, provided its CORS permissions allow reading its pixel data. It returns a promise that resolves with an array of predictions.
model.detect(video).then(function(predictions) {
console.log("Predictions:", predictions);
});

Prediction Format

The format of the predictions returned by model.detect is an array of objects containing the following properties:
  • class - (string) the name of the prediction; this will match one you chose during dataset creation.
  • bbox - (object) an object describing the predicted bounding box containing the following properties (each a float representing a number of pixels):
    • x - the x-coordinate of the center point.
    • y - the y-coordinate of the center point.
    • width - the width of the bounding box.
    • height - the height of the bounding box.
  • confidence - (float, 0-1) the certainty of the model in its prediction; a higher number means the model is more confident.
  • color - (string) a hex string representing the color of the bounding box (matching the color displayed in your dataset's annotation visualizations in Roboflow).
Example prediction:
[
{
"class": "hard-hat",
"bbox": {
x: 289.8,
y: 344.9,
width: 193.5,
height: 239.5
},
"confidence": 0.8083,
"color": "#F4004E"
},
{
"class": "hard-hat",
"bbox": {
x: 491.7,
y: 93.9,
width: 147.3,
height: 134.3
},
"confidence": 0.6982,
"color": "#F4004E"
}
]

Detailed Method and Property Definitions

The roboflow object is added to the global scope by the CDN. It has the following publicly accessible methods and properties:
  • roboflow.auth(options) - sets up the library to access your trained models using your publishable_key (accepted in an options object, eg {publishable_key:"<<YOUR KEY>>"}). There are no other publicly usable options for this method.
  • roboflow.load(options) - loads a model that was trained with Roboflow Train. Requires your model and version ids (accepted in an options object, eg {model:"your-model", version:1}). You may also include an onMetadata function in the options to receive information about your trained model from the API. Returns a Promise which resolves with the model object.
  • roboflow.VERSION identifies the version of roboflow.js that was loaded.
The model object returned by roboflow.load has the following methods and properties:
  • model.configure(options) - allows you to set prediction options (minimum confidence threshold, maximum box overlap, maximum number of objects, desired classes). See above for details.
  • model.getConfiguration() - returns the currently in-use configuration options.
  • model.getMetadata() - returns the metadata for the loaded model (name, type, icon, annotation group, input size, class list).
  • model.detect(input) - passes input (usually a <video>, <canvas>, or <img> element) to the model and gets predicted bounding boxes of target objects. Returns a Promise that resolves with an array of predictions (see above for the Prediction Format definition).
  • model.teardown() - destroys the model and cleans up its resources.
Previous
Mobile iOS (On Device)
Next
Changelog — roboflow.js
Last modified 3mo ago