Monday, May 7, 2012

How to decode QR code using Unity3D

UPDATE 2012-09-17: For decoding QR codes on Android devices, see here.

Hi,
here's how to decode QR code using Unity3D through camera. Please note that I'm using a library called ZXing (Zebra Crossing). I used the C# version since it is the most familiar language to me. I modified it a bit since Unity doesn't allow to use System.Drawing namespace.

To get access to the camera device I use WebCamTexture class which is new since Unity 3.5 (if I'm not mistaken). Here's my code in the Unity side:

using com.google.zxing.qrcode;

This is the only namespace to use from the dll if you just want to decode QR codes and nothing else.

WebCamTexture camTexture;
Thread qrThread;

void Start () {
    camTexture = new WebCamTexture();
    OnEnable();
    
    qrThread = new Thread(DecodeQR);
    qrThread.Start();
}

At first I'm declaring a new WebCamTexture to get the texture taken from the camera, and then I'm going to use another thread to process that texture.

int W, H, WxH;

void OnEnable () {
    if(camTexture != null) {
        camTexture.Play();
        W = camTexture.width;
        H = camTexture.height;
        WxH = W * H;
    }
}

void OnDisable () {
    if(camTexture != null) {
        camTexture.Pause();
    }
}

void OnDestroy () {
    qrThread.Abort();
    camTexture.Stop();
}

I use the OnEnable event handler to start the camera and retrieve it's size.

Color32[] c;
sbyte[] d;
int x, y, z;

void Update () {
    c = camTexture.GetPixels32();
}

void DecodeQR () {
    while(true) {
        try {
            d = new sbyte[WxH];
            z = 0;
            for(y = H - 1; y >= 0; y--) { // flip
                for(x = 0; x < W; x++) {
                    d[z++] = (sbyte)(((int)c[y * W + x].r)<<16 | ((int)c[y * W + x].g)<<8 | ((int)c[y * W + x].b));
                }
            }
            
            print(new QRCodeReader().decode(d, W, H).Text);
        }
        catch {
            continue;
        }
    }
}

Now, everytime a QR code is detected from the camera it will decode it. Nice and simple.
Here is the .dll file: link
Here is the .cs file: link

Thanks.

41 comments:

  1. Thanks for your work, Ariady!

    Your code is cool,which can implement camera-video decode with best performance.

    I've integrade zxing libruary by your post. But did not work out at last.

    I've read the origin zxing lib, it seems you have modified some important class like QRCodeReader.

    Now I am totally confused about this issue. about zxing. about texture flip.

    Could you offer some help?
    Or Could you send a copy of you modified zxing lib please?

    I'm afraid i can't find your email address. Could you leave an Email to me?

    Regards.

    Radar

    ReplyDelete
    Replies
    1. Hi Da L,
      yes I modified the Bitmap parts since Unity does not accept System.Drawing namespace.

      The modified Zxing library is provided at the end of my post, here's the link anyway:
      http://aurodeus.16mb.com/misc/zxing.zip

      If you need the modified version of Zxing Visual Studio project, it is located at:
      http://aurodeus.16mb.com/misc/zxingVSproj.zip

      I also provided the Unity sample project files:
      http://aurodeus.16mb.com/misc/QRunity.zip

      Lastly, the texture flipping process is required because the Color32 array from Unity is reversed vertically, meaning that the top most row of the image is located at the bottom most of the array. I hope it's clear enough...

      Delete
  2. Hey, thank you for your code :)


    Can you explain me this function?

    void DecodeQR () {
    while(true) {
    try {
    d = new sbyte[WxH];
    z = 0;
    for(y = H - 1; y >= 0; y--) {
    Debug.Log ("1. loop");
    for(x = 0; x < W; x++) {
    Debug.Log ("2. loop");
    d[z++] = (sbyte)(((int)c[y * W + x].r) << 16 | ((int)c[y * W + x].g) << 8 | ((int)c[y * W + x].b));
    }
    }
    Debug.Log(new QRCodeReader().decode(d, W, H).Text);
    }
    catch {
    Debug.Log ("failure");
    continue;
    }
    }
    }
    It is only once in the second loop. Witdh and hight are correct. Where is the failure?

    regards,
    Oli

    ReplyDelete
    Replies
    1. Hi Oliver Ebert,
      I'm not sure why it fails. I provide this Unity sample project files:
      http://aurodeus.16mb.com/misc/QRunity.zip

      The project contains:
      a scene, with a main camera, attached the qrCam script file, and the Zxing library itself.

      I have tested the project on my laptop in Unity Editor and it runs fine.

      Delete
    2. I try your code with Vuforia for iOS and Android (Augmented Reality Plugin for Unity) to combine. In your webcam solution is the first pixel left-buttom. On my solution ist the first pixel on right-top. How do I change the code so it works for me?

      Delete
    3. So, do you mean that the array that you got is like:
      2 1 0
      5 4 3
      8 7 6
      If that's the case then:
      - you don't need to flip it vertically
      - but flip it horizontally
      So, change the loops in the DecodeQR function to be like this:
      for(y = 0; y < H; y++) {
      for(x = W - 1; x >= 0; x++) {
      ...
      }
      }
      Because structure of the array needed for the decode funtion is like:
      0 1 2
      3 4 5
      6 7 8

      Delete
    4. do you have a chance to test your solution on an iPhone or Android device? I do not get it running. :(

      Delete
    5. Maybe I'm wrong but I think the solution can't work properly.

      The method new QRCodeReader().decode(d, W, H) uses RGBLuminanceSource which requires a sbyte array with red, green and blue values. The code above tries to assign the three channel values as an integer to a sbyte array. The sbyte cast should only return the blue value. The constructor of RGBLuminanceSource inside the method decode should be called with the fourth argument set to true (Is8Bit) and the calculation of the luminance value for the array d should be corrected to something similar like that:
      (sbyte)((r + g + g + b) >> 2)

      I can't test it myself at the moment because I don't have a running unity installation.

      Delete
    6. Actually, the bit shifting thing is also copied from the RGBLuminanceSource.cs file (http://code.google.com/p/zxing/source/browse/trunk/csharp/RGBLuminanceSource.cs) (line 108)
      I used that overloaded constructor and not the other (like the one you mentioned) cause I followed the Bitmap "route" (as if giving Zxing a Bitmap data through Unity).
      What I did was, I compared the Bitmap array in Visual Studio and the array from Unity using a small 5x6 pixel image file.
      And then I chose to follow the "Bitmap route" since I don't know what kind of byte array I should feed Zxing, I mean I don't know the structure of byte array I should give, like the pixel order and stuff. While in the sample app project I can simply see the values of Bitmap array.

      Delete
    7. reset W,H. because in ios webcam's width&height set when device ready perfect.

      and change for(...)
      like this

      for(y = 0; y <H; y++) {
      for(x = 0; x < W; x++) {
      }

      it will help u, thanks

      Delete
  3. I played a little around with unity and the sample source. But I didn't get access to my webcam with unity. Not sure what's wrong.

    Anyway I changed your code a little bit to show how I would realize it. Perhaps it works and perhaps it helps other people who want to implement a barcode scanner with unity.
    https://zxingnet.svn.codeplex.com/svn/trunk/Clients/UnityDemo
    It builds without errors but I can't test is because my webcam isn't activated with unity.

    Btw. I'm the creator of ZXing.Net which is an up-to-date port of the java original. With version 0.7.0.0 I added a binary for the use with unity which doesn't have a System.Drawing reference.

    It would be great if someone can give me a hint how I get my webcam running with unity. Do I need the pro version? Version number is 3.5.2f2.

    ReplyDelete
    Replies
    1. Oh wow, that's great! I will try your barcode scanner later... Thanks!

      The WebCamTexture class in unity doesn't need the pro version, I'm using the free version myself.

      Have you display the WebCamTexture object somewhere?

      Either directly on screen through OnGUI event or display it as a material texture of a plane object..

      You're using the very latest unity version right now..

      Delete
    2. I got it.
      I made some mistakes with namespaces which are not supported for behaviours I think.
      You can find my solution here:
      https://zxingnet.svn.codeplex.com/svn/trunk/Clients/UnityDemo

      It implements a custom luminance source for zxing and uses the BarcodeReader class instead of QRCodeReader. That means it decodes now every supported barcode.
      The luminance source calculates the luminance values directly from the Color32 structure. No need to create a temporary byte array.

      Delete
    3. Works this for Android an iOS?

      Delete
    4. Not sure. I didn't try it. But hopefully yes.

      Delete
    5. Michael i tested your code where you are using BarcodeReader class instead of QRCodeReader. It works fine for Unity Camera and i am able to decode QRCode using Unity but when i converted it for iOS and droid, it is not working. Can you please help me or anybody else who tried it.

      Delete
    6. Please provide some more information because I can't test it on that platforms. What error happens?

      Delete
    7. on iOS isn't any error message. Looks like, image isn't recognized or isn't interupted when QRcode is done via var result = barcodeReader.Decode(dummy, W, H);

      Delete
    8. I'm not sure if I can give help for iOS and Android because of missing environment.
      But I added a second sample for unity with vuforia support. Perhaps it would help you.
      https://zxingnet.svn.codeplex.com/svn/trunk/Clients/VuforiaDemo

      Delete
    9. Thx. Downloaded, but looks like missing unity demo scene

      Delete
    10. This comment has been removed by the author.

      Delete
    11. 80472/trunk/Clients/UnityDemo/Assets/UnityDemo.unity on iOS
      Debuger:
      ZXing_QrCode_Internal_FinderPatternFinder_find_System_Collections_Generic_IDictionary_2_ZXing_DecodeHintType_object:

      .byte 13,192,160,225,128,64,45,233,13,112,160,225,112,93,45,233,56,208,77,226,13,176,160,225,40,0,139,229,44,16,139,229
      .byte 44,0,155,229,0,0,80,227,12,0,0,10,44,0,155,229,3,16,160,227,44,32,155,229,0,32,146,229,0,128,159,229
      .byte 0,0,0,234
      .long mono_aot_zxing_unity_got - . + 304
      .byte 8,128,159,231,4,224,143,226,56,240,18,229,0,0,0,0,36,0,139,229,1,0,0,234,0,0,160,227,36,0,139,229
      .byte 36,0,155,229,0,0,203,229,40,0,155,229,8,16,144,229,1,0,160,225,0,224,145,229
      bl Lm_2d

      Delete
    12. Is that for me?
      Sorry, I'm not sure what I should do with the debugger output.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Great piece of code! I'm having some issues with iOS, though. Textures are flipped twice, and I'm still struggling with that piece of code.

    ReplyDelete
  6. I guess it's a minor optimization, buy you could create QRCodeReader just once by adding a member variable
    QRCodeReader QR;

    that is set to null in Start()

    and then, in the thread:

    if (QR == null)
    QR = new QRCodeReader();
    output = QR.decode(d, W, H).Text;

    being output a string variable that you may use to show the code in a label in OnGUI or something like that. But still struggling to make this work in iOS.

    ReplyDelete
  7. my mobile back camera didnt work this.
    because back camera focus-distance high.
    how can i handle webcam focus?

    sry for my bad english

    ReplyDelete
    Replies
    1. Right now there is no way to access camera focus using WebCamTexture class from Unity.

      You could use Vuforia (an AR plugin from Qualcomm for Unity) to access camera autofocus.
      You don't need to use WebCamTexture anymore if you're using Vuforia.

      Delete
  8. Hey -- I know this is a relatively old post now, but I've been trying for awhile to get this to work with Texture2D objects, and I can't get it to work. Even when I generated a code and then plugged it straight back into the decoder, I could not get it to decode -- it always fails. Is there anything about a Texture2D that I'm missing?

    Here is the class that I'm using:

    public class qrCam2 : MonoBehaviour {

    public Texture2D encoded;
    public string Text = "http://www.google.com/";
    void Start () {
    encoded = new Texture2D(1024, 1024);
    }

    void Update () {
    // ENCODE //
    com.google.zxing.common.ByteMatrix bytematrix = new QRCodeWriter().encode(Text, com.google.zxing.BarcodeFormat.QR_CODE, encoded.width, encoded.height);
    Color32[] col = new Color32[bytematrix.Width * bytematrix.Height];
    sbyte[] sCol = new sbyte[bytematrix.Width * bytematrix.Height];

    int x = 0;
    int y = 0;

    foreach (sbyte[] sarray in bytematrix.Array){
    foreach (sbyte s in sarray){

    sCol[1024 * y + x] = s;

    col[ ((y)*1024) + x ] = new Color32((byte)s, (byte)s, (byte)s, 255);

    x++;
    }
    x = 0;
    y++;
    }

    encoded.SetPixels32(col);
    encoded.Apply();



    // DECODE //
    try {

    print(new QRCodeReader().decode(sCol,bytematrix.Width, bytematrix.Height));

    }
    catch {

    print("fail");
    }
    }
    }

    The Texture2D "encoded" displays the proper QR code upside down, so sCol should should have the sbytes in the upside down order as well. But even with plugging that list of sbytes straight into the decode function it doesn't work. Any ideas?

    ReplyDelete
    Replies
    1. Hmm, the encoded Texture, have you tried to decode it using other app? does it decoded successfully? because I've never tried to encode.

      In my case, the result I got from Color32[] was inverted vertically, so I reordered the sbyte[] so that the top most row from the original Color32[] would be at the bottom most in the sbyte[].

      Delete
  9. First of all Thank you for you work and sharing this resource.
    Myself, I have a funny situation with it. When I run the app it works great - on the firsts decoding (at least in Editor, as I've tested it). It returning value from my QR code in Console, but even when I stop the app (or try to pause it) in Editor the function still returning the value from the QRcode. The camera is off, the Unity editor window is theoretically in stop mode, but it still working and returning value in the Console. Then when I try to play the app in editor again the Unity just hang.
    I'll try with your project and I let you know with the results.

    ReplyDelete
  10. Hi,

    My test on ipad says: ReaderException was throw and it's about GlobalHistogramBinarizer.estimateBlackPoint(System.Int32[] buckets)

    Callstack:
    common.GlobalHistogramBinarizer.estimateBlackPoint(Int32[] buckets)
    common.GlobalHistogramBinarizer.get_BlackMatrix()
    common.HybridBinarizer.binarizeEntireImage()
    common.HybridBinarizer.get_BlackMatrix()
    BinaryBitmap.get_BlackMatrix()
    qrcode.QRCodeReader.decode(BinaryBitmap image, ....)
    qrcode.QRCodeReader.decode(SByte[] d, Int32 W, Int32 H)
    qrCam.DecodeQR()

    Where I should start bugging?

    ReplyDelete
    Replies
    1. Hi,
      for mobile devices, I don't recommend using this piece of code. It will be very laggy.
      Please try the code for Android:
      http://ydaira.blogspot.com/2012/09/how-to-decode-qr-codes-using-unity3d.html
      I haven't tested it on any ios devices though. But it should be the same. It still uses Zxing.

      As for your error message, I'm not sure myself. I'm sorry, can't help there...

      Delete
    2. On second thought, I remember that screen resolution and camera resolution is not the same, at least on my device. You should throw in camera resolution, not screen resolution. It might be the problem. It may be not, I don't know. Goodluck.

      Delete
    3. Thank you for fast reply :)

      It's not problem if it be little laggy. Your better solution proposal was only for Android? My software need to run on ios. Is there way to work around that?

      Software is about scanning the qr codes and raporting them to the server. I think that I might do that so that software push .png -files from the camera to the server and then use libdecodeqr (http://trac.koka-in.org/libdecodeqr) from commandline to decode QRs. I also have thought using native code but then I have to do seperated versions for ios, android and standalone.

      What you mean throw in camera resolution? I'm using your example code and cameratexture's resolution:
      W = camTexture.width;
      H = camTexture.height;

      Delete
    4. Hi,

      yes I wrote there it's for Android, it's because I have not tested it on IOS, but it should run on IOS also, but I can't confirm that. Vuforia plugin also comes in Android and IOS version.

      If you still want to use the code from this page, I suggest to take a small portion of the area of image you want to process, because with full camera resolution it would be very slow on mobile devices. Unless you're planning to take a picture and then just process that one image.

      Since you're using camTexture width and height it should be okay. I mean, don't give the parameters like this:
      W = Screen.Width;
      H = Screen.Height;
      because screen resolution usually is not the same width camera resolution. Now I have no idea what error you encountered :-?

      Sorry for my bad english, I hope it's clear enough :D

      Delete
  11. hello,

    ZXing DLL file couldn't download

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. https://www.dllme.com/download.php?file=10723&id=b15314553420097274c6d86e667b3b86&key=ec658366ecebc643bbd5db62e27effbc28ee707dfbbf7ce832dd7b40f0101088&iss=1567797410

      Delete