Let me pull back the curtain a little for you and give you some insight into what the HTML5 Canvas API is all about and what it can do for you and your web apps.

In this article, I’m going to give an introductory overview of the HTML5 Canvas API.  The sample application that comes with this article, “Canvas Playground”, gives a number of canvas drawing examples and even lets you interactively play around with the Canvas API. Give it a try.

The HTML5 Canvas API is a procedural drawing API for JavaScript.  It’s well supported in all major browsers including Internet Explorer (starting with version 9) and is defined by the HTML5 Canvas W3C specification.  The Canvas API is a vector-based drawing API and should feel pretty familiar to those with exposure to graphics subsystems such as Quartz or GDI+.  But to those folks with less exposure the API may feel a bit overwhelming.  Here are just some of the capabilities at a glance:

• Defining Paths with lines, curves, arcs and more…
• Filling and Stroking Paths
• Clipping to a Path
• Filling and Stroking Text
• Affine Transformation (Translate, Scale, Rotate, Skew)
• Fills using Gradients and Patterns
• Pixel Manipulation
• Offscreen Rendering
• Rendering Blurred Shadows
• Dashed Lines, Line End Caps and Joins
• Exporting Canvas Rendering to a PNG or JPEG

If all of this sounds a little foreign to you don’t worry.  Coming up I’ll touch on a few of these capabilities and show some concrete examples to give you a feel for the API and how it works.  First of all the HTML5 Canvas is a browser DOM object and can be added to your webpage either declaratively through the <canvas> tag or programatically through the createElement method.  Once defined it contains a framebuffer bitmap at screen resolution that is the same width and height of the canvas object itself.

<canvas width=“640” height=“480”/>

Adding this tag to your HTML page will create a canvas object on your page that is 640 pixels wide by 480 pixels high and will be transparent.  The DOM object associated with the canvas object doesn’t directly have any drawing methods.  In order to actually draw on the canvas, you have to obtain a drawing context object by calling the canvas’s getContext method with a string parameter of “2d”.

context = canvas.getContext("2d");

The context object returned contains the actual drawing APIs that allow us to draw on our canvas object.  Let’s give it a try.  The following code snippet shows how to get a reference to a canvas object on a page, get a reference to it’s 2d context and draw a red filled rectangle on the canvas.

// Get a reference to the canvas object
canvas = document.querySelector("canvas");
// Get a reference to it’s 2d drawing context
context = canvas.getContext("2d");

// Displays a filled red rectangle on the screen
context.fillStyle="#FF0000";
context.fillRect(50,50,100,150);

Next lets look at drawing some more interesting shapes.  Outside of a few convenience methods for rectangles almost all shape drawing is done through the use of Paths.  So you might be asking yourself… What the Heck is a Path? A Path is a collection of lines or shapes (possibly disconnected) that once defined can be either filled or stroked.  Let’s look at a quick illustrative example.

// Define a path in the shape of a triangle with the moveTo and lineTo methods
context.beginPath();
context.moveTo(400,50);
context.lineTo(50,400);
context.lineTo(400,400);
context.closePath();

// Fill the path
context.fillStyle="purple";
context.fill();

// Stroke the path
context.strokeStyle="blue";
context.lineWidth=7;
context.stroke();

This code defines a path in the shape of a triangle and then goes on to fill the path and then to stroke that same path.  Let’s walk through this line by line.  The graphics context maintains a current path as a part of its state and calls to certain methods will accumulate either lines or shapes into the current path.  in the first line of code, a call to beginPath clears the current path from the context so that we start with a clean slate.  The context object also maintains a current “pen” position that is used to help define a path. This pen position can be changed with the moveTo method which simply sets the current positon of the “pen” to the provided x and y coordinates.  The lineTo method uses the current pen position as a starting point and the provided x and y coordinates to add a line to the current path.  The current pen position will be updated to the point at the end of the line. The next lineTo method call uses the current pen position and the provided x and y position to add another line to the current path. The next call to closePath will close the path by adding a line from the current pen position to the point that started the current path.  We have now defined a path that is in the shape of a triangle.  But it’s important to note that even though we have defined a path, nothing has been drawn to the canvas.  A path really just describes a set of lines and shapes.  Now that we have defined path, the path can be either filled or stroked to actually do the drawing.  The fill method will fill the interior of the path with the current fill style. By default the fill color or style is set to black. To specify a different style we can use any CSS color value. A few examples of valid color values: “red”, “purple”, rgb(255,255,0), rgba(255,255,0,128), #ff00ff. In addition to filling a path, you can also stroke the path which will outline the shapes defined in the path with the current stroke style which can also take any valid CSS color value. To change the width of the stroke, you can specify a pixel width for the stroked lines with the lineWidth property.  You can play around with these values in the Canvas Playground App by loading the “polygon sample”.  Next lets look at the “circle” example.

// Define a path in the shape of a circle with the arc method
context.beginPath();
context.arc(320, 240, 150, 0, 2*Math.PI);

// Fill the Path
context.fillStyle="orange";
context.fill();

// Stroke the Path
context.strokeStyle="green";
context.lineWidth=15;
context.stroke();

This example demonstates that paths aren’t limited to lines or polygonal shapes, but can also include arcs or circles.  My sample application goes on to show a few more slightly advanced examples and lets you interactively tweak the code.  Take a look at the full canvas specification and experiment.  If you liked learning a few basics about the HTML5 Canvas API but are hungry for more… This really is just the tip of the iceberg…   My upcoming ebook, “Essential Canvas for HTML5” will cover the subject in depth.  Just drop your email into the form below so that I can let you know when it’s available.

Take a look at my sample application as it provides a live demonstration of all the examples shown in this article as well as a few more slightly advanced ones. The full source code for the article can be downloaded here.

Learn the ins and outs of animation using the HTML5 Canvas API… Join my mailing list now so that find out when “Essential Canvas for HTML5” becomes available. Just drop your email into the form below!

Click here if you’d like to know more!

You want the users of your web app to be able to use their webcam to take their own profile shots and you’d like them to be able to personalize or stylize the picture before uploading it your weberver. Or maybe you’ve seen one of those fun photobooth-type apps that takes photos, applies cool image effects to them and lets you share them… wondering how they worked.

Imagine if your HTML5 app could programmatically access your webcam, take snapshots and modify the image data procedurally. You could then let your user’s upload or share their self portraits.

In this article, I’m going to teach you how to do that and more. Building on some articles that I’ve shared in the past, I’ll walk you through a sample application that displays the video from your webcam, applies image processing effects to that video, lets you take still snapshots of the filtered video and lets you serialize the snapshots so that they can be uploaded to a web server.

In one of my earlier articles, “Lights, Camera, Action!” I showed you the minimal amount of code needed to access you webcam and display the video into an HTML5 video object.  In this sample, we use the same code which lets us display the original video from the webcam.  But in order to do something more interesting with the video, we need to get the image data into a canvas object so that our Javascript code can manipulate it.  In fact in the sample app, we actually hide the HTML5 video object itself by setting it’s display style to none and the filtered video is actually rendered by a canvas object.  As it turns out a HTML5 video object can be used in place of a source image in a drawImage call to a 2d canvas context. Unfortunately the HTML5 video object doesn’t provide a frame accurate way of accessing the image data for a given frame. So in order to acquire the frame data we’ll leverage the requestAnimationFrame method to execute our callback on a recurring basis so that it can repeatedly acquire video frames. The following code is used to alias the vendor prefixed variants of requestAnimationFrame in order to make the code more portable across a range of browsers.  Credit: This requestAnimation polyfill was written by Paul Irish.

// Alias the vendor prefixed variants of requestAnimationFrame so that 
// we can access them via window.requestAnimationFrame fallback to 
// setTimeout at 60hz if not supported.
window.requestAnimationFrame = (function(){
  return  window.requestAnimationFrame       || 
          window.webkitRequestAnimationFrame || 
          window.mozRequestAnimationFrame    || 
          window.oRequestAnimationFrame      || 
          window.msRequestAnimationFrame     || 
          function( callback ){
            window.setTimeout(callback, 1000 / 60);
          };
})();

When requestAnimationFrame is invoked it will invoke the provided callback in approximately 1/60th of a second.  We could have also just used the setTimeout method.  But there are several advantages to using requestAnimationFrame over setTimeout.  The primary advantage is that if the browser tab is not currently visible or if our browser is minimized then requestAnimationFrame (unlike setTimeout) will not invoke the callback, thereby saving cpu cycles when our page is not visible.  Our requestAnimationFrame callback function is called update and is shown below:

function update(){  
  //...
  processVideoFrame();
  requestAnimationFrame(update); 
};

In the update function, processVideoFrame is called to acquire a single frame of video from the video object and process it.  A call to requestAnimationFrame is then made to queue our update function to be called again approximately 1/60th of a second later (Assuming that rate can be maintained by the CPU).  This function serves as the “pump” that will continually capture video frames and process them with any selected image filters.

function processVideoFrame() {
  // We have to check for the video dimensions here.
  // Dimensions will be zero until they can be determined from
  // the stream.
  if (context && video.videoWidth > 0 && video.videoHeight > 0) {
    // Resize the canvas to match the current video dimensions
    if (canvas.width != video.videoWidth) 
      canvas.width = video.videoWidth;
    if (canvas.height != video.videoHeight) 
      canvas.height = video.videoHeight;

    // Copy the current video frame by drawing it onto the 
    // canvas's context
    context.drawImage(video, 0, 0, canvas.width, canvas.height);
    processImage(canvas);
  }
}

In the processVideoFrame function, the canvas object is sized to match the dimensions of the video object. A call to the drawImage method will copy the pixel data from the current frame within the video object onto the canvas’s context.  A subsequent call to processImage is made to apply an image filter if one is currently selected.

function processImage() {
  if (canvas.width > 0 && canvas.height > 0) {
    if (imageFilter) {      
      context.putImageData(imageFilter.apply(null, [context.getImageData(0, 0, 
        canvas.width, canvas.height)]), 0, 0);
    }
  }
}

In another one of my articles, “How You Can Do Cool Image Effects Using HTML5 Canvas”, I demonstrated how to apply image filters to the pixel data contained within a canvas object. We’ll be doing the same thing here on each frame of captured video.  In the sample app, the following image filters can be applied to the video: grayscale, sepia, red, brighter, darker.  Please refer to my earlier article for more detail on image filters.

The next thing I want to cover is how you can take a snapshot of the image currently in our canvas, so that it can be uploaded or shared. The following function is used to take a snapshot.

function takeSnapshot() {
  var url = canvas.toDataURL();
  console.log(url);
  // Set the src of the image url to the data url
  document.querySelector("#snapshot").src = url;
  // Display the DOM elements that contain the snapshot
  document.querySelector("#snapshotdiv").style.display="inline-block";
}

The real work is done by the canvas’s toDataURL method.  This method takes the image data currently contained within the canvas, copies it, compresses it and returns a data URL of this image. So just what is a data URL?  A data URL (or more correctly… a data URI) is a way to represent a serialized chunk of data as a “stringified” URL.  An example is shown here:

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==

This example data URL encodes a PNG image of a red dot. To try this out for yourself, you can take this example and paste it into the URL bar of your browser and you’ll see a small red dot rendered. If you’d like to learn more about data urls, check out the wikipedia article.  Since this URL is a base64 encoded version of the image file itself. This string can be uploaded to your webserver and saved. It will just take a bit of string manipulation and a base64 decoder in your backend code to convert it back to a binary image file. Turning our attention back the takeSnapshot function, once we’ve obtained the data url for our image it can be assigned the src property of an image object to display the snapshot.

Get My Free Bonus Article on Color Matrix Filters!  I’ll send you a link to a bonus article for my data-driven Color Matrix Filter and sample application when you join my mailing list.  A Color Matrix Filter can do almost an infinite number of different color filter effects.  You won’t want to miss adding this to your toolchest!  Just drop your email in the form below:

That’s all for this go ‘round. Hope you’ve learned a few things and enjoyed the article. You can try out the sample app here and get the full source code here.

Argh!!… Your web application has to sort a lot of data and you get the dreaded error message… “A script on this page may be busy, or it may have stopped responding…” Or maybe you’re writing some image processing code that takes forever on large images… Your UI is non-responsive. What are you going to do!!?… You could try to break up the work into small pieces and use timers to spread out the work. But what a PITA!

What if JavaScript had the ability to run your code in the background and not interfere with the responsiveness of your UI. What if you could spawn a thread to do long running computations such as image filtering and sorting. Well with HTML5 Web Workers you can do just that! In the rest of this article, I’m going to teach you how to use Web Workers within your own applications. Click here to try the sample app.

Web Workers are an HTML5 feature that allow a JavaScript developer to create additional threads of execution which can be used to execute long running code in the background. Normally your JavaScript runs on what is called the main UI thread. Everything you do from JavaScript including all DOM manipulations are executed sequentially on this single thread. If your JavaScript code runs too long without returning control to the browser, your UI will stop responding to the user. The following single line of code shows how to create a web worker:

    
// A webworker is created and the JavaScript file longjob.js
// is loaded into its context.
worker = new Worker("longjob.js");

This code creates a new web worker object that has its own JavasScript context running in its own thread, loads the referenced JavaScript URL into that context and executes it.  In the next code block, we’ll walk through this again giving a bit more context.

var worker;

function runInWebWorker() {
  // Does our browser support webworkers?
  if (!worker && window.Worker) {
    // A webworker is created and the JavaScript file longjob.js
    // is loaded into its context.
    worker = new Worker("longjob.js");
    if (worker) {
      // Register an event handler that will receive messages from our
      // web worker
      worker.addEventListener('message', function(e) {
        // Our webworker sends us a message when it is done.
        alert("longJob returned: " + e.data);
      });    
    }
    // Send a message to the worker to execute the longJob function
    // in the context of the workers thread.
    worker.postMessage(30);
  }
  else alert("Web Workers are not Supported in this Browser.");
}

Message passing is the mechanism used to communicate between a web worker and the main UI thread.  The web worker’s postMessage method is used to send messages to and from the web worker.  In order to receive a message an event handler must be registered with the addEventListener method.  In the code block above, after checking to see if the browser supports web workers and we’ve created the web worker, a ‘message’ event handler is registered on the web worker object.  This particular event handler is used by the main UI thread to receive messages from our web worker.  Next a call to postMessage is made to send a message in the other direction to our web worker.  In the sample, we pass a simple numeric value of 30 as the message payload (which will be the number of seconds that we want our job to run).  Note: You can send more complicated messages in the form of arrays or simple objects.  In the next code block, we’ll look at the code loaded into our web worker, the contents of longjob.js:

// longjob.js
// function to simulate a long running job
// loops for approximately n seconds.
function longJob(n) {
  var start = new Date(); 
  var elapsedSeconds = 0;
  while(elapsedSeconds < n) {
    var v = 0;
    for (var i = 0; i < 1000000; i++) v += i;
    var end = new Date();
    elapsedSeconds = (end.getTime() - start.getTime())/1000.0;
  }
  return "All Done in " + elapsedSeconds + " seconds.";
}

// Our webworker registers for an message event so we can talk
// to it from our main thread and ask it to do something.
self.addEventListener('message', function(e) {

  // Invoke the longjob function within our worker thread
  // and pass in the parameter that was sent in.
  var result = longJob(e.data);

  //  Send a message back to the main thread with the result
  self.postMessage(result);
  }, false);

The longJob function simulates a long running function by taking a numeric value n and looping until n seconds have transpired.  Next an event handler is registered with a call to self.addEventListener so that our web worker can receive messages sent to it from the main UI thread.  Note that within the context of our web worker the special variables this and self reference the web worker object itself.  The parameter passed into the postMessage API is made available as a data property (shown as e.data in code above) on the event object passed into the event callback function.  When longJob completes our event handler sends a message back to the main thread using the postMessage method.

Web workers can be a very useful and empowering feature but they do have some limitations. All communication into and out of your web workers is done by passing in and out data in the form of messages.  All data is passed by value meaning that any object references are converted to a serialized copy of the objects data.  You can’t directly access or manipulate the DOM from a web worker.  While this may seem somewhat limiting these restrictions pretty much eliminate all of the synchronization bugs that can arise when using threads.

Want more on Web Workers? Signup for my mailing list and I’ll send you a link to the updated code from my earlier article, “How You Can Do Cool Image Effects Using HTML5 Canvas”, that will now run the image filters in the background using Web Workers. Just drop your email in the form below!

I hope you’ve enjoyed learning about web workers. You can get the full source code for the article here and try out the sample app here.

Wouldn’t it be great if you knew how to apply cool image effects such as these from JavaScript dynamically? Would you like to learn the code behind how these image filters work? You totally can… This article will give you the fundamentals for basic image processing using the HTML5 Canvas API. You’ll learn how to write code that accesses an image’s pixel data and transforms this pixel data programmatically to perform cool image effects. Click here to try the sample app.

The HTML 5 Canvas API is used here to access and manipulate the pixel data of an image. Once we have loaded an image, we can draw that image into the 2d context of a canvas object as the code snippet below shows.

    
    var image = new Image();
    image.onload = function () {
      if (image.width != canvas.width)
        canvas.width = image.width;
      if (image.height != canvas.height)
        canvas.height = image.height;
      context.clearRect(0, 0, canvas.width, canvas.height);
      context.drawImage(image, 0, 0, canvas.width, canvas.height);
      filterCanvas(imageFilter);
    }
    image.src = imageURL;
  }

Once the image has been drawn into the context, we can access the pixel data of the context by invoking the context’s getImageData method. This method will return an object that has information about every pixel in the image.

  var imageData = context.getImageData(0, 0, canvas.width, canvas.height);

The returned imageData object will contain a property called data which will reference a single dimension array of numbers. This array of numbers is a packed array containing a red, a green, a blue and an alpha value for each pixel in the image. Therefore the number of values in this array will be equal to W x H x 4. Where W is the width of the canvas, H is the height of the canvas and 4 is the number of color components (red, green, blue, alpha). The range for a color component value is from 0–255. If you try to assign a value to a color component outside of this range the value will be clamped. Using this array you can write code that inspects and manipulates the pixel data. To update the canvas’s context with any modifications to the imageData object the context’s putImageData method can be used to write the pixel data back.

       
  context.putImageData(imageData, 0, 0);

What you do with the imageData between calling these two methods is really what this article is all about and where you can apply a filter to transform the appearance of the image. So putting it all together as follows:

  
  // apply a filter to the image data contained in the canvas object
  function filterCanvas(filter) {
    if (canvas.width > 0 && canvas.height > 0) {
      var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
      filter(imageData);
      context.putImageData(imageData, 0, 0);
    }
  }

The first filter that I’ll describe is often referred to as a grayscale filter. This filter sets the intensity of the red, green and blue color components to the arithmetic average of those values. This has the effect of changing a color image into a black and white or grayscale image. If you look at the snippet below you’ll see that the code loops through 1 pixel each time thru the loop. The value referenced by d[i] will always be the red component for the current pixel. The value referenced by d[i+1] will always be the green component. The value referenced by d[i+2] will always be the blue component. The value referenced by d[i+3] will always be the alpha component for the current pixel. Note: The alpha value is used to specify the transparency of the pixel. In all of the filters shown in this article the alpha component is not referenced or altered; therefore leaving it intact. The grayscale filter adds the red, green and blue values together and divides by three yielding an average value. This average value is assigned back to all 3 color components. Give this a try by clicking the grayscale link in the sample app.

  
  // grayscale filter using an arithmetic average of the color 
  // components
  grayscale = function (pixels, args) {
    var d = pixels.data;
    for (var i = 0; i < d.length; i += 4) {
      var r = d[i];
      var g = d[i + 1];
      var b = d[i + 2];
      d[i] = d[i + 1] = d[i + 2] = (r+g+b)/3;
    }
    return pixels;
  };

The next effect I’ll talk about is another popular one often referred to as a sepia filter. A sepia filter imparts a warm, antique look to an image. The math behind a sepia filter leverages a number of weighted formulas for each color component that factor in a contribution from each color component to arrive at a new value. Looking at the formulas in the code snippet below you’ll see a number of “magic” weights that will give us the desired effect. Each time through the loop the red, green and blue components for a single pixel will be updated. An interesting exercise for the reader is to experiment with these weights to come with your own  unique filter.

  
  // sepia-style filter that gives a warm antique feel to an image
  sepia = function (pixels, args) {
    var d = pixels.data;
    for (var i = 0; i < d.length; i += 4) {
      var r = d[i];
      var g = d[i + 1];
      var b = d[i + 2];
      d[i]     = (r * 0.393)+(g * 0.769)+(b * 0.189); // red
      d[i + 1] = (r * 0.349)+(g * 0.686)+(b * 0.168); // green
      d[i + 2] = (r * 0.272)+(g * 0.534)+(b * 0.131); // blue
    }
    return pixels;
  };

The next filter has a lot of similarities to the grayscale filter. But rather than applying the average back to all color components it only applies it to the red component thereby shifting all of the color information to red. The green and blue channels are zeroed out. Another excercise for the reader is to make a blue or a green filter.

  
  // filter that shifts all color information to red
  red = function (pixels, args) {
    var d = pixels.data;
    for (var i = 0; i < d.length; i += 4) {
      var r = d[i];
      var g = d[i + 1];
      var b = d[i + 2];
      d[i] = (r+g+b)/3;        // apply average to red channel
      d[i + 1] = d[i + 2] = 0; // zero out green and blue channel
    }
    return pixels;
  };

The brightness filter is the last one we’ll cover in this article. In this filter a positive or negative value is added to each red, green and blue component. This has the effect of either brightening the image given a positive value or darkening the image with a negative value. The function for this filter is slightly different than the others in that we use a JavaScript closure to create a parameterized function. By invoking the brightness function with a numeric parameter a new function instance is returned that will bound to the supplied delta parameter. This newly returned function object is the actual filter function.

  
  // filter that brightens an image by adding a fixed value 
  // to each color component
  // a javascript closure is used to parameterize the filter 
  // with the delta value
  brightness = function(delta) {
      return function (pixels, args) {
        var d = pixels.data;
        for (var i = 0; i < d.length; i += 4) {
          d[i] += delta;     // red
          d[i + 1] += delta; // green
          d[i + 2] += delta; // blue   
        }
        return pixels;
      };
  };

Want to learn more about image filters…

Get My Free Bonus Article on Color Matrix Filters!  I’ll send you a link to a bonus article for my data-driven Color Matrix Filter and sample application when you join my mailing list.  A Color Matrix Filter can do almost an infinite number of different color filter effects.  You won’t want to miss adding this to your toolchest!  Just drop your email in the form below:

I hope you’ve enjoyed this brief tour on image filters.  You can get the full source code for the article here and try out the sample app here.

So You Want to Access Your Webcam from Javascript?

Web Real-Time Communications or WebRTC is an exciting new HTML5 technology that not only gives JavaScript the ability to do peer-to-peer (P2P) audio/video conferencing and P2P data communications between browsers, it also lets you have programmatic access to attached cameras and microphones. Prior to WebRTC you had to rely on a browser plugin such as Adobe Flash to be able to do similar things. But recently there has been a big push to add these capabilites as intrinsic features to web browsers with Google really leading the charge and through standardization efforts by the W3C WebRTC working group. It is a brave new world in that only some browsers have added WebRTC support and even those that do can have subtle (and some not so subtle) differences in both interface and implementation. But more and more you’re seeing real applications appearing that are using WebRTC in very cool and interesting ways. One great example of WebRTC in action is a PhotoBooth-like app called WebCam Toy. This application should give you a sense of the possibilites that WebRTC is opening up.

In the rest of this article, I’m going to walk you through the minimum code needed to get your webcam to display video in your browser. The latest versions of Chrome or Firefox are recommended for running the code included in this article and for experimenting with WebRTC. If you already have one of these browsers you can try out my example.

So lets get started…

WebRTC defines the getUserMedia method on the top level navigator object as the mechanism for getting access to an attached camera/microphone. Since WebRTC is still in a state of flux, we do have to shim a number of vendor prefixed variants to increase the portability of our code as follows:

// Alias the vendor prefixed variants of getUserMedia 
// so we can access them via navigator.getUserMedia
navigator.getUserMedia = navigator.getUserMedia ||  
    navigator.webkitGetUserMedia || navigator.mozGetUserMedia ||    
    navigator.msGetUserMedia;

After checking to make sure that the getUserMedia method is available it can be invoked as follows:

if (navigator.getUserMedia) {
    // Call the getUserMedia method here
    navigator.getUserMedia({video: true}, successCallback, 
        failureCallback);
}

This will cause the browser to prompt the user for permission to access an attached camera. Note: This is done for security reasons so that a random website can’t access your camera without your permission. If there is an attached camera and the user grants access then the function passed in as successCallback in the above example will get invoked; otherwise the failureCallback function will be called. Note: Unfortunately at the time of this writing you can not determine whether the failure was due to the lack of an attached camera or whether the user denied access. On success, the successCallback function will be passed a stream object that be used to stream the video data from the camera to the video object that cooresponds to our video tag.

// Alias the vendor prefixed variants of the URL object 
// so that we can access them via window.URL
window.URL = window.URL || 
    window.webkitURL || window.mozURL || window.msURL;

// This function will be called if a webcam is available 
// and the user has granted access for the web 
// application to use it.
function successCallback(stream) {
    if (video.mozSrcObject !== undefined) {
        video.mozSrcObject = stream;
    } else {
        video.src = (window.URL && 
            window.URL.createObjectURL(stream)) ||
            stream;
    };
    video.play();
}

WebRTC leverages the HTML5 <video> tag as the DOM element for rendering video content from your camera. If you look at the example you’ll see that we included the video tag with no attributes in the body of our document.

<video/>

Now that we have a reference to the video stream. We need to connect it up to the video object defined by the video tag so we can see the video. Things get a little messy here because of some interface differences between browsers. Most browsers allow you to set the src property to a URL generated from the createObjectURL method but Firefox defines a different property called mozSrcObject that allows you to pass the stream object in directly. The code snippet shown above shows how to deal with this minor inconvenience. Once we’ve hooked up the stream to the video object, we simply have to call play on the video object to start rendering the video.

That’s all there is to it to get the output of your camera working in your browser with WebRTC. Very very cool…

All in all WebRTC is off to a great start for further enhancing the web. But there are a number of unfortunate and annoying limitations that hopefully get addressed in future enhancements to WebRTC. Currently you can’t enumerate the set of supported resolutions for a given camera and its not really possibly to cleanly probe for supported resolutions. However if you do know the set of resolutions supported by your camera you can specify them by passing in additional constraints to the getUserMedia method. As an example:

navigator.getUserMedia({video: {mandatory: {minWidth: 1024, 
    minHeight: 768 }}}, 
    successCallback, failureCallback);

Again you can get the full source code for this article here and I hope you have a great time exploring the new frontier that is WebRTC!

Join my mailing list to get my tips, tricks and articles delivered straight to your inbox.

There I said it… I mean lets face it, building apps in HTML+CSS is pretty much like doing word processing in LaTeX.  What ever happened to decent visual tools? And I’m tired of dealing with weird quirky layout issues.

Don’t get me wrong HTML for building apps is definitely (eh) OK and it’s been proven that you can create applications at scale with it.  But I think you end up working around quirks in layout behavior way too much.  HTML was originally designed for making non-wysiwyg “rich” text documents. The ability to build apps is bolted on and feels like it.

Also I can’t help but wonder will the pinnacle of visual application design tools be Interface Builder or Delphi.  I don’t think we’ve really moved forward in terms of the visual tools used to build apps since the advent of these tools.  Rich Internet Application frameworks and plugins like Flash and Silverlight (along with FlexBuilder and Blend) tried to solve this within the browser.  But it’s looking more and more like these are going to fizzle out of existence.

So What to Do?

I think the core issue is that there is too much layout policy baked into the browser.  The included layout primitives (and their paradigms) don’t really do enough and in a lot of ways just get in the way.  And its not like workable application layout isn’t a solved problem.  There are no signs of layout primitives like springs and struts, dock-based layout or flex box layout.  Don’t get me wrong these things should not be baked into the HTML rendering of the browser.  Browsers should work more towards baking in less policy and favor more composible and programmable lower layers.  It turns out that HTML5 already has the right pieces.  Basically we just need a way to express visually what we need to express with minimal policy.  These are in the form of Canvas2d and WebGL.  Why not develop UI frameworks completely on these programmable foundations.  Just some examples of some interesting work that’s been done here and here.

The W3C should really focus on the edge.  HTML apps continue to inch closer and closer to being able to offer the richness of experience that a native app can.  The gaps really come down to insufficient edge interfaces and insufficient integration with the platform that the browser is running on.  Instead of focusing on enhancing HTML’s layout capabilities the W3C should focus on innovations at the edge (some illustrative examples) such as defining a decent programmable audio pipeline for input and output or a programmable pipeline for video decode.

Here are a few more edge problems that i’d like to see solved that currently limit how well applications completely built on top of Canvas2d and WebGL would fare compared to apps being built using HTML.

• Virtual Keyboard management – The ability to programmatically detect and invoke virtual keyboards and events related to screen geometry changes when these have been invoked.
• Accessibility/Screen Readers – The ability to expose accessible virtual objects and their metadata (eg. alternate text) to screen readers.
• Input Method Editors – for inputting international text
• Better clipboard and drag and drop support
• Better geometry and orientation change events
• Debugging - Exposing virtual DOM objects built as Javascript objects to visual debugging tools (in Chrome for example)

I mean there are SO many things to love from Google… Chrome, Picasa, Maps, Search… But the problem is that we’re again seeing fragmentation limiting what the Web could become and perhaps even threatening its long term survival.

Google seems to be shifting it’s mission away from let’s “organize the world‘s information and make it universally accessible and useful” to one of let’s make “high impact products—the ones that improve the lives of billions of people“.  Maybe Larry fancies himself being the second coming of Jobs… and I’ll be the first to say…

NOT A BAD GOAL TO HAVE!!

The EVIL IS that they don’t seem to mind subverting the web to make it all happen.  Google’s apparent “we’ll go it alone” mindset pushing initiatives like DART and SPDY… and their continued product positioning of Chrome As The Platform (chrome as opposed to the web)… are threatening the very fabric of the Web.

And I think folks might be underestimating the threat… Whilst Chrome doesn’t have the monopoly browser position that IE once had to drive this fragmentation… To paraphrase what a friend of mine recently blogged about… people won’t “buy” browsers but they will “buy” browsers+services…

Trust us you’ll work better with Chrome… GWEB anybody?

Joe just might get his wish.

Give it a try.