This article is sponsored by Adobe.

---

Advertisements

Interested in Flex graphics? Check out the new book Flex 4 Fun by Chet Haase, available as a PrePrint™ from Artima Press.

The advent of multicore processors has been one of the most significant computer architecture developments in recent years. Chip-level parallelism provides a computing environment with low communication overhead between compute nodes. That low communication cost, in turn, allows developers to use parallel computing techniques and algorithms in more places than was possible with traditional parallel or multiprocessing computers.

In addition to CPUs with multiple compute cores, recent computing devices contain a heterogeneous set of processors: Server-class hardware may have multiple general-purpose CPUs—and each of those processors may have multiple cores—while desktop and mobile devices routinely feature dedicated graphics processors (GPU) and audio processing hardware.

Although the ready availability of multicore CPUs and heterogeneous processors has generated excitement among developers, there is also some trepidation: benefiting from multicore processor capabilities requires the use of specialized programming techniques and libraries. In addition, computations should ideally execute on the most suitable type of processor available in a device. Integer arithmetic, for instance, may be best handled by a general-purpose CPU, while floating point operations may be well-suited for the GPU. Scheduling such processing has to take into account not only execution times, but processor caches and, in mobile devices, estimated power consumption as well.

GPU Parallelism

Fortunately, a new generation of libraries and runtimes aim to simplify the task of making the most of available compute resources. Adobe's Flash Player, for example, is increasingly becoming a runtime that supports both heterogeneous processors and parallel computing. The latest version, Flash Player 10.1, delegates some graphics processing to GPUs on desktops, notebooks, and especially on mobile devices. The ability to delegate processing to a GPU, in turn, allows application code to take advantage of the highly parallel architectures of graphics chips.

As computer graphics became increasingly part and parcel of user interfaces and desktop windowing systems, algorithm designers improved graphics performance mainly by perfecting pipeline-based rendering techniques. Pipeline-based graphics rendering can exploit dataflow parallelism: a "pipeline" takes some input, such as a series of pixel values, performs some transformations on each pixel value of that input, and produces an output. Because the same transformation can often be applied to each pixel in an image, several graphics pipelines can be set up to execute in parallel, each pipeline handling a portion of the input data. GPUs are designed to execute graphics pipelines very fast and often in parallel.

GPUs are not restricted to performing graphics rendering tasks, however, and can be used for general-purpose programming as well. Unlike ubiquitous CPU architectures, such as the Intel x86, no single graphics processor architecture emerged as a de facto standard. Instead of a homogeneous chip architecture, several graphics programming models developed to exploit GPU programming at a higher level, such as OpenGL and Microsoft's Direct3D. Hardware manufacturers typically provide optimized implementations of these standards for their chips that exploit special hardware features, including parallel execution.

Pixel Shaders

A common concept to high-level pipeline-based graphics programming models, such as OpenGL, is the pixel shader: Introduced in the popular RenderMan graphics software, a pixel shader is small program that can take one or more images as inputs, as well as possibly a complex array of input parameters, and perform some operation, or a series of operations, on every pixel of the input image, producing some output data. The underlying runtime executing a pixel shader is free to optimize execution, including executing rendering operations in parallel and taking advantage of the pipelined architectures of modern GPUs.

Pixel shader programs can be written not only to operate on pixels in image data, but to operate on any data input, such as a series of numeric values. Pixel shaders, therefore, afford developers a relatively straightforward way to benefit from the parallel computing capabilities of graphics processors at a high abstraction level: you can focus on writing a serial program that defines a simple transformation function, and the underlying runtime takes care of parallelizing your program, if feasible.

If no dedicated graphics processor is available, the runtime implementation is clever enough to just run your pixel shader on the general-purpose CPU. By the same token, a well-designed graphics runtime can discover and take advantage of even more sophisticated GPUs, such as those that include multiple processing cores.

Pixel Bender

Adobe Labs recently introduced Pixel Bender, a freely available development tool for writing pixel shaders. Pixel Bender is a language, an IDE, a compiler, and associated tools that allow developers to define pixel transformation operations in Pixel Bender programs—Pixel Bender kernels,—execute those programs on input images, and export those programs in a binary format. The binary format Pixel Bender produces can execute not only in Flash Player, but also in Adobe graphics programs, such as PhotoShop and AfterEffects.

Pixel Bender's most obvious application is for writing image filters. Graphics editing programs routinely provide a collection of standard image filters, such as drop shadows, blurs, sharpening, beveling, and so on. With Pixel Bender, anyone familiar with Pixel Bender's simple language can create new image filters.

Pixel Bender kernels are not restricted to image filters, however: Pixel Bender can help create any serial programs suitable for highly parallel, pipelined execution on GPUs. As Flash Player versions increasingly make use of graphics hardware, Pixel Bender programs will likely benefit from increased parallelism.

Because image filters provide a straightforward illustration of using Pixel Bender with Flex, the rest of this article focuses on a Flex application that applies a filter to an image. Note, however, that Flex filters can be applied not only to images, but to any graphics object, including user interface components. Filters are also fundamental to Flex effects: Effects often involve animation applied to Flex filter properties.

Although the Pixel Bender language is fairly easy to learn, Adobe set up a Pixel Bender kernel exchange that currently contains hundreds of freely available Pixel Bender filters. This article will use one such filter: Zoom Blur, created by Ryan Phelan. As its name suggests, Zoom Blur performs a zoom effect on an input image, and then applies a blur, creating a result similar to a motion blur:

Pixel Bender Kernels in Flex

A handful of new ActionScript classes in the flash.display package facilitate interaction with Pixel Bender kernels from Flex. A Pixel Bender kernel is represented in Flex by the Shader class. A Shader can load an external Pixel Bender kernel's bytecode and make the kernel's operations, as well as input and output, available to any ActionScript or MXML code.

Input images to a Shader are specified via one or more ShaderInput values. Input can come in the form of a BitmapData, a ByteArray, or a Vector of Numbers. Flex's optional static typing allows for an easy-to-use API when specifying kernel input data. A Shader defines a public data property, which is just a generic object. Kernel input image names then become available as properties of the data object.

The following ActionScript code snippet loads an image from a URL, and makes that image's pixel data available as an input to a Pixel Bender kernel:

var zoomBlurShader:Shader = new Shader();

var loader:Loader = new Loader();
loader.contentLoaderInfo.addEventListener(Event.COMPLETE, onImageLoaded);
loader.load(new URLRequest("http://myimage.png");

function onImageLoaded(event: Event):void {
  zoomBlurShader.data.myInputImage = event.target.content.bitmapData;  
}

Non-image parameters to a Pixel Bender kernel can be specified with ShaderParameter properties. The Zoom Blur kernel consumes three input parameters: the amount of zoom, and the vertical and horizontal amount to apply during the blur. Specifying the zoom amount, for instance, can be accomplished thus:

zoomBlurShader.data.amount.value[0] = 0.08;

Once a Shader is defined, it can be used to create a ShaderFilter. Like any Flex image filter, ShaderFilter can be applied to any Flex DisplayObject, not only images. The filter will be applied as soon as the component's filters property is assigned:

var shaderFilter:ShaderFilter = new ShaderFilter(zoomBlurShader);
myDisplayComponent.filters = [shaderFilter];

As in many Flex applications, a combination of MXML and ActionScript affords elegant and concise code. The following is the entire source code for this article's example:

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009"
               xmlns:s="library://ns.adobe.com/flex/spark">

  <fx:Script>
    <![CDATA[                        
      import spark.filters.ShaderFilter;
                                                                                        
      [Embed(source="zoomBlur.pbj", mimeType="application/octet-stream")]
      private var ShaderClass:Class;

      private var shader:Shader = new Shader(new ShaderClass());  

      [Bindable]
      private var filter:ShaderFilter = new ShaderFilter(shader);

      [Bindable]
      [Embed("cathedral.png")]
      private var cathedral:Class;
                                                                                
      private function updateImage():void {                     
         shader.data.amount.value[0] = amnt.value;
         filter.shader = shader;   
      }
    ]]>
  </fx:Script>
                                
  <s:HGroup verticalAlign="middle">
    <s:BitmapImage id="im" source="{cathedral}" filters="{[filter]}"/>    
    <s:Label text="Zoom:"/>
    <s:HSlider id="amnt" 
                   minimum="0" snapInterval="0.005" 
                   maximum="0.5" liveDragging="true"
                   change="updateImage()"/>
  </s:HGroup>
</s:Application>

Once the application loads, it creates a Shader instance based on the Pixel Bender kernel's embedded bytecode. It is also possible to load the kernel code from a network location instead of directly embedding it in the Flex application. The Shader is then used as a parameter to construct a ShaderFilter.

The main user interface component is a BitmapImage instance with a source image embedded in this example. The image could just as easily be loaded from a network location via a Loader. The second significant component is an HSlider. As the user drags the slider, the slider fires change events. The updateImage() method is registered as a handler for those change events.

updateImage() simply assigns the slider's current value to the shader's amount parameter. Because the ShaderFilter's shader property is cached by the filter instance, we need to re-assign the shader with the changed values to the filter. In Flex 4, changes occuring to filters assigned to an image are immediately applied to the image.

Working with Bitmap Images

The above example illustrates image manipulation using the high-level Pixel Bender toolkit and Flex's BitmapImage class. The Flash Player API provides lower-level imaging functions as well. Flash Player can display both bitmap and vector images. While bitmap images can be embedded in Flex applications with the Embed directive, as in the above example, or loaded at runtime from a URL, vector images, such as SVG content, can be used only when embedded.

Bitmap images can be in JPG, GIF, or PNG formats. At the heart of the Flash Player's low-level bitmap operations is the Bitmap class that represents a bitmap image. Once instantiated, a Bitmap provides a reference to the actual bit-level image data via its bitmapData property of type BitmapData. The BitmapData class, in turn, defines a large number of methods to access and manipulate the image's each pixel, such as adjusting the various color channels' values or the alpha value. Such methods provide complete flexibility in terms of image editing, but they may not benefit from many runtime optimizations available to Pixel Bender filters.

Summary

With each subsequent version, Adobe's Flash Player provides an increasing sophisticated array of optimizations to improve the performance of Flex applications. Some of those optimizations take advantage of dedicated graphics hardware on the client, allowing for very fast image operations. Such operations, or filters, can be applied not only to traditional images, such as photos, but also to any Flex user interface component. Efficient runtime imaging operations, when combined with animations, in turn, enable developers to take advantage of effects that enhance a user's experience, such as smooth transitions, user-friendly runtime shadings, or reflections.

Acknowledgements

The author would like thank Adobe's Chet Haase for his comments on this article.

Resources

Adobe's Flash Builder 4
http://labs.adobe.com/technologies/flashbuilder4

Flex 4 SDK
http://opensource.adobe.com/wiki/display/flexsdk/Flex+SDK

Flex.org
http://www.flex.org

Chet Haase's video showing how to use Pixel Bender effects
http://tv.adobe.com/watch/codedependent/pixel-bender-shaders-and-flex-4

Pixel Bender
http://labs.adobe.com/downloads/pixelbender.html

Pixel Bender Exchange
http://www.adobe.com/cfusion/exchange/index.cfm?event=productHome&exc=26

Talk back!

Have an opinion? Be the first to post a comment about this article.

About the author

Frank Sommers is Editor-in-Chief of Artima.