Unity Barracuda
The Barracuda package is a lightweight cross-platform neural network inference library for Unity. Barracuda can run neural networks on both the GPU and CPU.
The Barracuda neural network import pipeline is built on the ONNX (Open Neural Network Exchange) format, which lets you bring in neural network models from a variety of external frameworks, including Pytorch, TensorFlow, and Keras.
Barracuda is a simple, developer-friendly API for neural network execution. You can start using Barracuda with as little code as this:
var model = ModelLoader.Load(filename);
var engine = WorkerFactory.CreateWorker(model, WorkerFactory.Device.GPU);
var input = new Tensor(1, 1, 1, 10);
var output = engine.Execute(input).PeekOutput();
Download the model file from here Download
Import the above model to the asset folder, the model is imported and shows up as an asset of type NNModel. Create a new c# script and attach the script to a GameObject and add references to the following
public NNModel modelAsset;
private Model _runtimeModel;
private IWorker _engine;
The NNModel is the model reference of the ONNX file.A model is an asset wrapper. It stores the model in binary format. Load and compile it into a run-time model. The core interface into the Barracuda engine is called Worker. A Worker breaks down the model into executable tasks and schedules them on the GPU or CPU.
At the Start Unity event load the model and create a worker engine. Use CPU for android build and GPU for windows/ Macos.
void Start()
{
_runtimeModel = ModelLoader.Load(modelAsset);
_engine = WorkerFactory.CreateWorker(_runtimeModel, WorkerFactory.Device.CPU);
}
Create a prediction struct to handle the output of the model. The output is a list of float values. Based on the struct we create a base class
[Serializable]
public struct Prediction
{
public int predictedValue;
public List < float > predictedValues;
public void SetPrediction(Tensor tensor)
{
predictedValues = tensor.AsFloats().ToList();
predictedValue = predictedValues.IndexOf(predictedValues.Max());
}
}
When you have loaded the model and created the IWorker interface, you can execute the model. You can provide inputs either as a sole Tensor object (if the model has a single input) or as a dictionary of name and Tensor pairs.
Example for Single Input Scenario -
Tensor input = new Tensor(batch, height, width, channels);
worker.Execute(input);
You can create a Tensor from an input Texture or save a Tensor to a Texture directly. You can directly pass Texture2D, Texture2DArray, Texture3D or RenderTexture to Barracuda without accessing individual pixels on the CPU. If the model has a single output, you can use worker.PeekOutput() . Otherwise, provide output names. worker.PeekOutput() returns the reference to the output tensor without transferring ownership to you. Calling worker. PeekOutput() guarantees that there are no additional memory allocations. However, it means that tensor values are lost after the next call to worker.Execute() or after the call to worker.Dispose().
int channelCount = 1;
var input = new Tensor(texture, channelCount);
Tensor output = _engine.Execute(input).PeekOutput();
prediction.SetPrediction(output);
for (var i = 0; i < prediction.predictedValues.Count; i++)
{
var value = prediction.predictedValues[i];
text.text += $" {i} = {value} \n";
}
output.Dispose();
Download the Test Script from here Download