Java example source code file (UTF8Test.java)

This example Java source code file (UTF8Test.java) is included in the alvinalexander.com "Java Source Code Warehouse" project. The intent of this project is to help you "Learn Java by Example" ^TM.

Learn more about this Java project at its project page.

Java - Java tags/keywords

arrayreference, breakpointevent, exception, expected, failed, jdi, set, string, stringbuffer, stringreference, testscaffold, unsupportedencodingexception, utf8targ, utf8test, utf\-8, util

The UTF8Test.java Java example source code

/*
 * Copyright (c) 2004, 2007, Oracle and/or its affiliates. All rights reserved.
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 *
 * This code is free software; you can redistribute it and/or modify it
 * under the terms of the GNU General Public License version 2 only, as
 * published by the Free Software Foundation.
 *
 * This code is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
 * version 2 for more details (a copy is included in the LICENSE file that
 * accompanied this code).
 *
 * You should have received a copy of the GNU General Public License version
 * 2 along with this work; if not, write to the Free Software Foundation,
 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
 *
 * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
 * or visit www.oracle.com if you need additional information or have any
 * questions.
 */

/**
 *  @test
 *  @bug 5033550
 *  @summary  JDWP back end uses modified UTF-8
 *
 *  @author jjh
 *
 *  @run build TestScaffold VMConnection TargetListener TargetAdapter
 *  @run compile -g UTF8Test.java
 *  @run main UTF8Test
 */

/*
  There is UTF-8 and there is modified UTF-8, which I will call M-UTF-8.
  The two differ in the representation of binary 0, and
  in some other more esoteric representations.
  See
      http://java.sun.com/developer/technicalArticles/Intl/Supplementary/#Modified_UTF-8
      http://java.sun.com/javase/6/docs/technotes/guides/jni/spec/types.html#wp16542

  All the following are observations of the treatment
  of binary 0.  In UTF-8, this represented as one byte:
      0x00

  while in modified UTF-8, it is represented as two bytes
      0xc0 0x80

  ** I haven't investigated if the other differences between UTF-8 and
     M-UTF-8 are handled in the same way.

 Here is how these our handled in our BE, JDWP, and FE:

 - Strings in .class files are M-UTF-8.

 - To get the value of a string object from the VM, our BE calls
      char * utf = JNI_FUNC_PTR(env,GetStringUTFChars)(env, string, NULL);
   which returns M-UTF-8.

- To create a string object in the VM, our BE VirtualMachine.createString() calls
      string = JNI_FUNC_PTR(env,NewStringUTF)(env, cstring);
      This function expects the string to be M-UTF-8
      BUG:  If the string came from JDWP, then it is actually UTF-8

- I haven't investigated strings in JVMTI.

- The JDWP spec says that strings are UTF-8.  The intro
  says this for all strings, and the createString command and
  the StringRefernce.value command say it explicitly.

- Our FE java writes strings to JDWP as UTF-8.

- BE function outStream_writeString uses strlen meaning
  it expects no 0 bytes, meaning that it expects M-UTF-8
  This function writes the byte length and then calls
  outStream.c::writeBytes which just writes the bytes to JDWP as is.

  BUG: If such a string came from the VM via JNI, it is actually
       M-UTF-8
  FIX:  - scan string to see if contains an M-UTF-8 char.
          if yes,
             - call String(bytes, 0, len, "UTF8")
               to get a java string.  Will this work -ie, the
               input is M-UTF-8 instead of real UTF-8
             - call some java method (NOT JNI which
               would just come back with M-UTF-8)
               on the String to get real UTF-8


- The JDWP StringReference.value command does reads a string
  from the BE out of the JDWP stream and does this to
  createe a Java String for it (see PacketStream.readString):
         String readString() {
          String ret;
          int len = readInt();

          try {
              ret = new String(pkt.data, inCursor, len, "UTF8");
          } catch(java.io.UnsupportedEncodingException e) {

  This String ctor converts _both- the M-UTF-8 0xc0 0x80
  and UTF-8 0x00  into a Java char containing 0x0000

  Does it do this for the other differences too?

Summary:
1.  JDWP says strings are UTF-8.
    We interpret this to mean standard UTF-8.

2.  JVMTI will be changed to match JNI saying that strings
    are M-UTF-8.

3.  The BE gets UTF-8 strings off JDWP and must convert them to
    M-UTF-8 before giving it to JVMTI or JNI.

4.  The BE gets M-UTF-8 strings from JNI and JVMTI and
    must convert them to UTF-8 when writing to JDWP.


 Here is how the supplementals are represented in java Strings.
 This from java.lang.Character doc:
    The Java 2 platform uses the UTF-16 representation in char arrays and
    in the String and StringBuffer classes. In this representation,
    supplementary characters are represented as a pair of char values,
    the first from the high-surrogates range, (\uD800-\uDBFF), the second
    from the low-surrogates range (\uDC00-\uDFFF).
  See utf8.txt


----

NSK Packet.java in the nsk/share/jdwp framework does this to write
a string to JDWP:
 public void addString(String value) {
        final int count = JDWP.TypeSize.INT + value.length();
        addInt(value.length());
        try {
            addBytes(value.getBytes("UTF-8"), 0, value.length());
        } catch (UnsupportedEncodingException e) {
            throw new Failure("Unsupported UTF-8 ecnoding while adding string value to JDWP packet:\n\t"
                                + e);
        }
    }
 ?? Does this get the standard UTF-8?  I would expect so.

and the readString method does this:
        for (int i = 0; i < len; i++)
            s[i] = getByte();

        try {
            return new String(s, "UTF-8");
        } catch (UnsupportedEncodingException e) {
            throw new Failure("Unsupported UTF-8 ecnoding while extracting string value from JDWP packet:\n\t"
                                + e);
        }
Thus, this won't notice the modified UTF-8 coming in from JDWP .


*/

import com.sun.jdi.*;
import com.sun.jdi.event.*;
import com.sun.jdi.request.*;
import java.io.UnsupportedEncodingException;
import java.util.*;

    /********** target program **********/

/*
 * The debuggee has a few Strings the debugger reads via JDI
 */
class UTF8Targ {
    static String[] vals = new String[] {"xx\u0000yy",           // standard UTF-8 0
                                         "xx\ud800\udc00yy",     // first supplementary
                                         "xx\udbff\udfffyy"      // last supplementary
                                         // d800 = 1101 1000 0000 0000   dc00 = 1101 1100 0000 0000
                                         // dbff = 1101 1011 1111 1111   dfff = 1101 1111 1111 1111
    };

    static String aField;

    public static void main(String[] args){
        System.out.println("Howdy!");
        gus();
        System.out.println("Goodbye from UTF8Targ!");
    }
    static void gus() {
    }
}

    /********** test program **********/

public class UTF8Test extends TestScaffold {
    ClassType targetClass;
    ThreadReference mainThread;
    Field targetField;
    UTF8Test (String args[]) {
        super(args);
    }

    public static void main(String[] args)      throws Exception {
        new UTF8Test(args).startTests();
    }

    /********** test core **********/

    protected void runTests() throws Exception {
        /*
         * Get to the top of main()
         * to determine targetClass and mainThread
         */
        BreakpointEvent bpe = startToMain("UTF8Targ");
        targetClass = (ClassType)bpe.location().declaringType();
        targetField = targetClass.fieldByName("aField");

        ArrayReference targetVals = (ArrayReference)targetClass.getValue(targetClass.fieldByName("vals"));

        /* For each string in the debuggee's 'val' array, verify that we can
         * read that value via JDI.
         */

        for (int ii = 0; ii < UTF8Targ.vals.length; ii++) {
            StringReference val = (StringReference)targetVals.getValue(ii);
            String valStr = val.value();

            /*
             * Verify that we can read a value correctly.
             * We read it via JDI, and access it directly from the static
             * var in the debuggee class.
             */
            if (!valStr.equals(UTF8Targ.vals[ii]) ||
                valStr.length() != UTF8Targ.vals[ii].length()) {
                failure("     FAILED: Expected /" + printIt(UTF8Targ.vals[ii]) +
                        "/, but got /" + printIt(valStr) + "/, length = " + valStr.length());
            }
        }

        /* Test 'all' unicode chars - send them to the debuggee via JDI
         * and then read them back.
         */
        doFancyVersion();

        resumeTo("UTF8Targ", "gus", "()V");
        try {
            Thread.sleep(1000);
        } catch (InterruptedException ee) {
        }


        /*
         * resume the target listening for events
         */

        listenUntilVMDisconnect();

        /*
         * deal with results of test
         * if anything has called failure("foo") testFailed will be true
         */
        if (!testFailed) {
            println("UTF8Test: passed");
        } else {
            throw new Exception("UTF8Test: failed");
        }
    }

    /**
     * For each unicode value, send a string containing
     * it to the debuggee via JDI, read it back via JDI, and see if
     * we get the same value.
     */
    void doFancyVersion() throws Exception {
        // This does 4 chars at a time just to save time.
        for (int ii = Character.MIN_CODE_POINT;
             ii < Character.MIN_SUPPLEMENTARY_CODE_POINT;
             ii += 4) {
            // Skip the surrogates
            if (ii == Character.MIN_SURROGATE) {
                ii = Character.MAX_SURROGATE - 3;
                break;
            }
            doFancyTest(ii, ii + 1, ii + 2, ii + 3);
        }

        // Do the supplemental chars.
        for (int ii = Character.MIN_SUPPLEMENTARY_CODE_POINT;
             ii <= Character.MAX_CODE_POINT;
             ii += 2000) {
            // Too many of these so just do a few
            doFancyTest(ii, ii + 1, ii + 2, ii + 3);
        }

    }

    void doFancyTest(int ... args) throws Exception {
        String ss = new String(args, 0, 4);
        targetClass.setValue(targetField, vm().mirrorOf(ss));

        StringReference returnedVal = (StringReference)targetClass.getValue(targetField);
        String returnedStr = returnedVal.value();

        if (!ss.equals(returnedStr)) {
            failure("Set: FAILED: Expected /" + printIt(ss) +
                    "/, but got /" + printIt(returnedStr) + "/, length = " + returnedStr.length());
        }
    }

    /**
     * Return a String containing binary representations of
     * the chars in a String.
     */
     String printIt(String arg) {
        char[] carray = arg.toCharArray();
        StringBuffer bb = new StringBuffer(arg.length() * 5);
        for (int ii = 0; ii < arg.length(); ii++) {
            int ccc = arg.charAt(ii);
            bb.append(String.format("%1$04x ", ccc));
        }
        return bb.toString();
    }

    String printIt1(String arg) {
        byte[] barray = null;
        try {
             barray = arg.getBytes("UTF-8");
        } catch (UnsupportedEncodingException ee) {
        }
        StringBuffer bb = new StringBuffer(barray.length * 3);
        for (int ii = 0; ii < barray.length; ii++) {
            bb.append(String.format("%1$02x ", barray[ii]));
        }
        return bb.toString();
    }

}