NAME¶

SYNOPSIS¶

 use Unicode::Japanese;
 use Unicode::Japanese qw(unijp);
 
 # convert utf8 -> sjis
 
 print Unicode::Japanese->new($str)->sjis;
 print unijp($str)->sjis; # same as above.
 
 # convert sjis -> utf8
 
 print Unicode::Japanese->new($str,'sjis')->get;
 
 # convert sjis (imode_EMOJI) -> utf8
 
 print Unicode::Japanese->new($str,'sjis-imode')->get;
 
 # convert zenkaku (utf8) -> hankaku (utf8)
 
 print Unicode::Japanese->new($str)->z2h->get;

DESCRIPTION¶

FEATURES¶

•

An instance of Unicode::Japanese internally holds a string in UTF-8.

•

This module is implemented in two ways: XS and pure perl. If efficiency is important for you, you should build and install the XS module. If you don't want to, or if you can't build the XS module, you may use the pure perl module instead. In that case, only you have to do is to copy Japanese.pm into somewhere in @INC.

•

This module can convert characters from zenkaku (full-width) form to hankaku (half-width) form, and vice versa. Conversion between hiragana (one of two sets of japanese phonetical alphabet) and katakana (another set of japanese phonetical alphabet) is also supported.

•

This module has mapping tables for emoji (graphic characters) defined by various japanese mobile phones; DoCoMo i-mode, ASTEL dot-i and J-PHONE J-Sky. Those letters are mapped on Unicode Private Use Area so unicode strings it outputs are still valid even if they contain emoji, and you can safely pass them to other softwares that can handle Unicode.

•

This module can map some emoji from one set to another. Different mobile phones define different sets of emoji, so mapping each other is not always possible. But since some emoji exist in two or more sets with similar appearance, this module considers those emoji to be the same.

•

This module uses the mapping table for MS-CP932 instead of the standard Shift_JIS. The Shift_JIS encoding used by MS-Windows (MS-SJIS/MS-CP932) slightly differs from the standard.

•

When the module converts strings from Unicode to Shift_JIS, EUC-JP or ISO-2022-JP, unicode letters which can't be represented in those encodings will be encoded in "&#dddd;" form (decimal character reference). Note, however, that letters in Unicode Private Use Area will be replaced with '?' mark ('QUESTION MARK'; U+003F) instead of being encoded. In addition, encoding to character sets for mobile phones makes every unrepresentable letters being '?' mark.

•

On perl-5.8.0 or later, this module handles the UTF-8 flag: the method utf8() returns UTF-8 byte string, and the method getu() returns UTF-8 character string.

 

Currently the method get() returns UTF-8 byte string but this behavior may be changed in the future.

 

Methods like sjis(), jis(), utf8(), and such like return byte string. new(), set(), getcode() methods just ignore the UTF-8 flag of strings they take.

REQUIREMENT¶

•

perl 5.10.x, 5.8.x, etc. (5.004 and later)

•

(optional) C Compiler. This module supports both XS and Pure Perl. If you have no C Compilers, Unicode::Japanese will be installed as Pure Perl module.

•

(optional) Test.pm and Test::More for testing.

METHODS¶

$s = Unicode::Japanese->new($str [, $icode [, $encode]])

Create a new instance of Unicode::Japanese.

 

Any given parameters will be internally passed to the method "set"().

$s = unijp($str [, $icode [, $encode]])

Same as Unicode::Jananese->new(...).

$s->set($str [, $icode [, $encode]])

 auto
 utf8 ucs2 ucs4
 utf16-be utf16-le utf16
 utf32-be utf32-le utf32
 sjis cp932 euc euc-jp jis
 sjis-imode sjis-imode1 sjis-imode2
 utf8-imode utf8-imode1 utf8-imode2
 sjis-doti sjis-doti1
 sjis-jsky sjis-jsky1 sjis-jsky2
 jis-jsky  jis-jsky1  jis-jsky2
 utf8-jsky utf8-jsky1 utf8-jsky2
 sjis-au sjis-au1 sjis-au2
 jis-au  jis-au1  jis-au2
 sjis-icon-au sjis-icon-au1 sjis-icon-au2
 euc-icon-au  euc-icon-au1  euc-icon-au2
 jis-icon-au  jis-icon-au1  jis-icon-au2
 utf8-icon-au utf8-icon-au1 utf8-icon-au2
 ascii binary

$str = $s->get

$str = $s->getu

$code = $s->getcode($str)

$code = $s->getcodelist($str)

$str = $s->conv($ocode, $encode)

$s->tag2bin

Interpret decimal character references (&#dddd;) in the instance, and replaces them with single characters they represent.

$s->z2h

Replace zenkaku (full-width) letters in the instance with hankaku (half-width) letters.

$s->h2z

Replace hankaku (half-width) letters in the instance with zenkaku (full-width) letters.

$s->hira2kata

Replace any hiragana in the instance with katakana.

$s->kata2hira

Replace any katakana in the instance with hiragana.

$str = $s->jis

$str: byte string in ISO-2022-JP

 

Get the internal string of instance with encoding it in ISO-2022-JP.

$str = $s->euc

$str: byte string in EUC-JP

 

Get the internal string of instance with encoding it in EUC-JP.

$str = $s->utf8

$str: byte string in UTF-8

 

Get the internal UTF-8 string of instance.

 

On perl-5.8.0 or later, the UTF-8 flag of resulting string is turned off.

$str = $s->ucs2

$str: byte string in UCS-2

 

Get the internal string of instance as a sequence of raw UCS-2 letters in big-endian. Note that this is different from UTF-16BE as raw UCS-2 sequence has no concept of surrogate pair.

$str = $s->ucs4

$str: byte string in UCS-4

 

Get the internal string of instance as a sequence of raw UCS-4 letters in big-endian. This is practically the same as UTF-32BE.

$str = $s->utf16

$str: byte string in UTF-16

 

Get the insternal string of instance with encoding it in UTF-16 in big-endian with no BOM prepended.

$str = $s->sjis

$str: byte string in Shift_JIS

 

Get the internal string of instance with encoding it in Shift_JIS (MS-SJIS / MS-CP932).

$str = $s->sjis_imode

$str: byte string in 'sjis-imode'

 

Get the internal string of instance with encoding it in 'sjis-imode'.

$str = $s->sjis_imode1

$str: byte string in 'sjis-imode1'

 

Get the internal string of instance with encoding it in 'sjis-imode1'.

$str = $s->sjis_imode2

$str: byte string in 'sjis-imode2'

 

Get the internal string of instance with encoding it in 'sjis-imode2'.

$str = $s->sjis_doti

$str: byte string in 'sjis-doti'

 

Get the internal string of instance with encoding it in 'sjis-doti'.

$str = $s->sjis_jsky

$str: byte string in 'sjis-jsky'

 

Get the internal string of instance with encoding it in 'sjis-jsky'.

$str = $s->sjis_jsky1

$str: byte string in 'sjis-jsky1'

 

Get the internal string of instance with encoding it in 'sjis-jsky1'.

$str = $s->sjis_jsky

$str: byte string in 'sjis-jsky'

 

Get the internal string of instance with encoding it in 'sjis-jsky'.

$str = $s->sjis_icon_au

$str: byte string in 'sjis-icon-au'

 

Get the internal string of instance with encoding it in 'sjis-icon-au'.

$str_arrayref = $s->strcut($len)

$len = $s->strlen

$len: character width of the internal string

 

Calculate the character width of the internal string. Half-width characters have width of one unit, and full-width characters have width of two units.

$s->join_csv(@values);

@values: array of strings

 

Build a line of CSV from the arguments, and store it into the instance. The resulting line has a trailing line break ("\n").

@values = $s->split_csv;

@values: array of strings

 

Parse a line of CSV in the instance and return each columns. The line will be chomp()ed before getting parsed.

 

If the internal string was decoded from 'binary' encoding (see methods new() and set()), the UTF-8 flags of the resulting array of strings are turned off. Otherwise the flags are turned on.

SUPPORTED ENCODINGS¶

 +---------------+----+-----+-------+
 |encoding       | in | out | guess |
 +---------------+----+-----+-------+
 |auto           : OK : --  | ----- |
 +---------------+----+-----+-------+
 |utf8           : OK : OK  | OK    |
 |ucs2           : OK : OK  | ----- |
 |ucs4           : OK : OK  | ----- |
 |utf16-be       : OK : --  | ----- |
 |utf16-le       : OK : --  | ----- |
 |utf16          : OK : OK  | OK(#) |
 |utf32-be       : OK : --  | OK    |
 |utf32-le       : OK : --  | OK    |
 |utf32          : OK : --  | OK(#) |
 +---------------+----+-----+-------+
 |sjis           : OK : OK  | OK    |
 |cp932          : OK : OK  | ----- |
 |euc            : OK : OK  | OK    |
 |euc-jp         : OK : OK  | ----- |
 |jis            : OK : OK  | OK    |
 +---------------+----+-----+-------+
 |sjis-imode     : OK : OK  | OK    |
 |sjis-imode1    : OK : OK  | ----- |
 |sjis-imode2    : OK : OK  | ----- |
 |utf8-imode     : OK : OK  | ----- |
 |utf8-imode1    : OK : OK  | ----- |
 |utf8-imode2    : OK : OK  | ----- |
 +---------------+----+-----+-------+
 |sjis-doti      : OK : OK  | OK    |
 |sjis-doti1     : OK : OK  | ----- |
 +---------------+----+-----+-------+
 |sjis-jsky      : OK : OK  | OK    |
 |sjis-jsky1     : OK : OK  | ----- |
 |sjis-jsky2     : OK : OK  | ----- |
 |jis-jsky       : OK : OK  | ----- |
 |jis-jsky1      : OK : OK  | ----- |
 |jis-jsky2      : OK : OK  | ----- |
 |utf8-jsky      : OK : OK  | ----- |
 |utf8-jsky1     : OK : OK  | ----- |
 |utf8-jsky2     : OK : OK  | ----- |
 +---------------+----+-----+-------+
 |sjis-au        : OK : OK  | OK    |
 |sjis-au1       : OK : OK  | ----- |
 |sjis-au2       : OK : OK  | ----- |
 |jis-au         : OK : OK  | ----- |
 |jis-au1        : OK : OK  | ----- |
 |jis-au2        : OK : OK  | ----- |
 |sjis-icon-au   : OK : OK  | ----- |
 |sjis-icon-au1  : OK : OK  | ----- |
 |sjis-icon-au2  : OK : OK  | ----- |
 |euc-icon-au    : OK : OK  | ----- |
 |euc-icon-au1   : OK : OK  | ----- |
 |euc-icon-au2   : OK : OK  | ----- |
 |jis-icon-au    : OK : OK  | ----- |
 |jis-icon-au1   : OK : OK  | ----- |
 |jis-icon-au2   : OK : OK  | ----- |
 |utf8-icon-au   : OK : OK  | ----- |
 |utf8-icon-au1  : OK : OK  | ----- |
 |utf8-icon-au2  : OK : OK  | ----- |
 +---------------+----+-----+-------+
 |ascii          : OK : --  | OK    |
 |binary         : OK : OK  | ----- |
 +---------------+----+-----+-------+
 (#): guessed when it has bom.

GUESSING ORDER¶

 1.  utf32 (#)
 2.  utf16 (#)
 3.  utf32-be
 4.  utf32-le
 5.  ascii
 6.  jis
 7.  sjis-jsky (pp)
 8.  euc
 9.  sjis
 10. sjis-jsky (xs)
 11. sjis-au
 12. sjis-imode
 13. sjis-doti
 14. utf8
 15. unknown

DESCRIPTION OF UNICODE MAPPING¶

Shift_JIS

This module uses the mapping table of MS-CP932.

 

<ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT>

 

When the module tries to convert Unicode string to Shift_JIS, it represents most letters which isn't available in Shift_JIS as decimal character reference ('&#dddd;'). There is one exception to this: every graphic characters for mobile phones are replaced with '?' mark.

 

For variants of Shift_JIS defined for mobile phones, every unrepresentable characters are replaced with '?' mark unlike the plain Shift_JIS.

EUC-JP/ISO-2022-JP

This module doesn't directly convert Unicode string from/to EUC-JP or ISO-2022-JP: it once converts from/to Shift_JIS and then do the rest translation. So characters which aren't available in the Shift_JIS can not be properly translated.

DoCoMo i-mode

This module maps emoji in the range of F800 - F9FF to U+0FF800 - U+0FF9FF.

ASTEL dot-i

This module maps emoji in the range of F000 - F4FF to U+0FF000 - U+0FF4FF.

J-PHONE J-SKY

The encoding method defined by J-SKY is as follows: first an escape sequence "\e\$" comes to indicate the beginning of emoji, then the first byte of an emoji comes next, then the second bytes of at least one emoji comes next, then "\x0f" comes last to indicate the end of emoji. If a string contains a series of emoji whose first bytes are identical, such sequence can be compressed by cascading second bytes of them to the single first byte.

 

This module considers a pair of those first and second bytes to be one letter, and map them from 4500 - 47FF to U+0FFB00 - U+0FFDFF.

 

When the module encodes J-SKY emoji, it performs the compression automatically.

AU

This module maps AU emoji to U+0FF500 - U+0FF6FF.

PurePerl mode¶

   use Unicode::Japanese qw(PurePerl);

BUGS¶

•

This module doesn't directly convert Unicode string from/to EUC-JP or ISO-2022-JP: it once converts from/to Shift_JIS and then do the rest translation. So characters which aren't available in the Shift_JIS can not be properly translated.

•

The XS implementation of getcode() fails to detect the encoding when the given string contains \e while its encoding is EUC-JP or Shift_JIS.

•

Japanese.pm is composed of textual perl script and binary character conversion table. If you transfer it on FTP using ASCII mode, the file will collapse.

SUPPORT¶

    perldoc Unicode::Japanese

•

AnnoCPAN: Annotated CPAN documentation

 

http://annocpan.org/dist/Unicode-Japanese <http://annocpan.org/dist/Unicode-Japanese>

•

CPAN Ratings

 

http://cpanratings.perl.org/d/Unicode-Japanese <http://cpanratings.perl.org/d/Unicode-Japanese>

•

RT: CPAN's request tracker

 

http://rt.cpan.org/NoAuth/Bugs.html?Dist=Unicode-Japanese <http://rt.cpan.org/NoAuth/Bugs.html?Dist=Unicode-Japanese>

•

Search CPAN

 

http://search.cpan.org/dist/Unicode-Japanese <http://search.cpan.org/dist/Unicode-Japanese>

CREDITS¶

COPYRIGHT & LICENSE¶