Updating WPF image from camera frame grabber thread

I have created a WPF application in .NET 4.6.1. to display video from the frame grabbing event for a machine vision camera.

In XAML there is an image control bound to a WriteableBitmap

<Image x_Name="CameraImage" Grid.Column="1" HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Source="{Binding CameraImage}"/>

In the View Model:

public event PropertyChangedEventHandler PropertyChanged;
protected void NotifyPropertyChanged([CallerMemberName] string propertyName = null)
{
    //Console.WriteLine(propertyName);
    PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
}

private WriteableBitmap cameraImage;
private IntPtr cameraImagePtr;
public WriteableBitmap CameraImage
{
    get { return cameraImage; }
    set
    {
        cameraImage = value;
        cameraImagePtr = cameraImage.BackBuffer;
        NotifyPropertyChanged();
    }
}

The WriteableBitmap is initialised as so:

CameraImage = new WriteableBitmap(2448, 2048, 0, 0, PixelFormats.Bgr24, null);

In the grab event generated by the camera we have:

// Occurs when an image has been acquired and is ready to be processed.
private void OnImageGrabbed(Object sender, ImageGrabbedEventArgs e)
{
    try
    {
        // Get the grab result.
        IGrabResult grabResult = e.GrabResult;

        // Check if the image can be displayed.
        if (grabResult.IsValid)
        {
            byte[] buffer = grabResult.PixelData as byte[];

            converter.OutputPixelFormat = PixelType.BGR8packed;
            lock(cameraImage)
            {
                // Converts camera Bayer array image to BGR image and saves at cameraImagePtr
                converter.Convert(cameraImagePtr, 2448 * 2048 * 3, grabResult);
            }                    

            // Make WriteableBitmap dirty (on UI thread) so that the WPF Image control redraws (parent is UI object)
            parent.Dispatcher.Invoke(new Action(() =>
            {
                cameraImage.Lock();
                cameraImage.AddDirtyRect(new Int32Rect(0, 0, cameraImage.PixelWidth, cameraImage.PixelHeight));
                cameraImage.Unlock();
            }), DispatcherPriority.Render);
        }
    }
    catch (Exception exception)
    {
        Console.WriteLine(exception.Message);
        throw exception;
    }
    finally
    {
        // Dispose the grab result if needed for returning it to the grab loop.
        e.DisposeGrabResultIfClone();
    }
}

converter is Basler.Pylon.PixelDataConverter. It converts the grabbed frame which is in Bayer array format into a normal BRG image.

The lock is in place because there is another routine to save the current image, and I don’t want to do this while it is being updated:

    public void SaveImage()
    {
        if (cameraImage == null) return;
        lock(cameraImage)
        {
            String filename = folder + "\" + stem + "-" + set.ToString("D2") + "-" + subset.ToString("D2") + ".png";
            if (filename != string.Empty)
            {
                using (FileStream stream = new FileStream(filename, FileMode.Create))
                {
                    PngBitmapEncoder encoder = new PngBitmapEncoder();
                    encoder.Frames.Add(BitmapFrame.Create(cameraImage));
                    encoder.Save(stream);
                }
            }
        }
    }

Questions

• I can’t just pass cameraImage.BackBuffer to the writing (Converter.convert()) function, I have to do cameraImagePtr = cameraImage.BackBuffer; and then use cameraImagePtr. Why is that?

Answer: Because another thread owns the cameraImage object

• The fans on my laptop spin up when this is running – it seems like it is using a lot of CPU. Is there a more efficient way? From what I can tell I am not making extra copies of the frame.

Answer: On another PC the CPU is only 18%, so not sure why the laptop fan is going so hard.

Update

The camera images are quite large – 2448 x 2048 x 3 bytes.

As is, I get 20 fps (camera true fps is 30) with CPU usage of 21%. There is frame tearing occuring.

I did some changes to the code with the following results:

1. Moved converter.Convert(...) into the Invoke block. FPS dropped to 14 and the UI was a tiny bit more unresponsive. Frame tearing stopped.

2. Removed the lock. No difference.

3. Create a temporary byte array and pass it to converter.Convert(...) instead of cameraImagePtr, then inside the Invoke block use ‘cameraImage.WritePixels(…)’ to copy to the WriteableBitmap on the UI thread. No tearing, but fps dropped to 14.

4. Same as above, but using BeginInvoke instead. Fps back up to 20 but no tearing, and now the lock can be removed!

So it seems option 4 is the best, the code is now:

if (grabResult.IsValid)
{
    converter.OutputPixelFormat = PixelType.BGR8packed;
    byte[] bgrBuffer = new byte[2448 * 2048 * 3];
    converter.Convert(bgrBuffer, grabResult);

    parent.Dispatcher.BeginInvoke(new Action(() =>
    {
        cameraImage.Lock();
        cameraImage.WritePixels(new Int32Rect(0, 0, 2448, 2048), bgrBuffer, 2448 * 3, 0);
        cameraImage.AddDirtyRect(new Int32Rect(0, 0, cameraImage.PixelWidth, cameraImage.PixelHeight));
        cameraImage.Unlock();
    }), DispatcherPriority.Render);
    frames++;
}

1) Are you sure that it is safe to write to cameraImagePtr from background thread the way you do it? Because documentation clearly states that it is not:

Update the back buffer only between calls to the Lock and Unlock
methods. If you do not follow the Lock/Unlock workflow described in
the WriteableBitmap class remarks, undefined behaviors, such as
tearing, can occur.

I suggest you convert your image to separate byte[] buffer and then use WriteableBitmap.WritePixels in-between lock/unlock to block-copy the pixel data.

2) The way you use lock statement looks fishy. What exactly are you locking and why? If events come on different threads and you can’t handle them fast enough, then you have to come up with a throttling mechanism, that would drop some of the camera frames. Otherwise events will just keep piling up on this lock in no particular order and eat up your CPU (which is what you observe).

3) Depending on what else is going on in your app, Dispatcher.Invoke call can be pretty expensive performance-wise. Dispatcher.BeginInvoke on the other hand is much cheaper, but you will have to be really careful with your buffers, so that background threads do not overwrite the buffer, that is currently being rendered on UI thread.