UTF8Handler implements Unicode aware operations for strings, these operations will be used by the Chars proxy when $KCODE is set to ‘UTF8’.
- capitalize
- compose
- compose_codepoints
- consumes?
- decompose
- decompose_codepoints
- downcase
- g_length
- g_pack
- g_unpack
- in_char_class?
- index
- insert
- lstrip
- normalize
- reorder_characters
- reverse
- rstrip
- size
- slice
- strip
- tidy_bytes
- to_case
- translate_offset
- u_unpack
- upcase
| HANGUL_SBASE | = | 0xAC00 |
| Hangul character boundaries and properties | ||
| HANGUL_LBASE | = | 0x1100 |
| HANGUL_VBASE | = | 0x1161 |
| HANGUL_TBASE | = | 0x11A7 |
| HANGUL_LCOUNT | = | 19 |
| HANGUL_VCOUNT | = | 21 |
| HANGUL_TCOUNT | = | 28 |
| HANGUL_NCOUNT | = | HANGUL_VCOUNT * HANGUL_TCOUNT |
| HANGUL_SCOUNT | = | 11172 |
| HANGUL_SLAST | = | HANGUL_SBASE + HANGUL_SCOUNT |
| HANGUL_JAMO_FIRST | = | 0x1100 |
| HANGUL_JAMO_LAST | = | 0x11FF |
| UNICODE_WHITESPACE | = | [ (0x0009..0x000D).to_a, # White_Space # Cc [5] <control-0009>..<control-000D> 0x0020, # White_Space # Zs SPACE 0x0085, # White_Space # Cc <control-0085> 0x00A0, # White_Space # Zs NO-BREAK SPACE 0x1680, # White_Space # Zs OGHAM SPACE MARK 0x180E, # White_Space # Zs MONGOLIAN VOWEL SEPARATOR (0x2000..0x200A).to_a, # White_Space # Zs [11] EN QUAD..HAIR SPACE 0x2028, # White_Space # Zl LINE SEPARATOR 0x2029, # White_Space # Zp PARAGRAPH SEPARATOR 0x202F, # White_Space # Zs NARROW NO-BREAK SPACE 0x205F, # White_Space # Zs MEDIUM MATHEMATICAL SPACE 0x3000, # White_Space # Zs IDEOGRAPHIC SPACE ].flatten.freeze |
| All the unicode whitespace | ||
| UNICODE_LEADERS_AND_TRAILERS | = | UNICODE_WHITESPACE + [65279] |
| BOM (byte order mark) can also be seen as whitespace, it‘s a non-rendering character used to distinguish between little and big endian. This is not an issue in utf-8, so it must be ignored. | ||
| UTF8_PAT | = | /\A(?: [\x00-\x7f] | [\xc2-\xdf] [\x80-\xbf] | \xe0 [\xa0-\xbf] [\x80-\xbf] | [\xe1-\xef] [\x80-\xbf] [\x80-\xbf] | \xf0 [\x90-\xbf] [\x80-\xbf] [\x80-\xbf] | [\xf1-\xf3] [\x80-\xbf] [\x80-\xbf] [\x80-\xbf] | \xf4 [\x80-\x8f] [\x80-\xbf] [\x80-\xbf] )*\z/xn |
| Borrowed from the Kconv library by Shinji KONO - (also as seen on the W3C site) | ||
| UNICODE_TRAILERS_PAT | = | /(#{codepoints_to_pattern(UNICODE_LEADERS_AND_TRAILERS)})+\Z/ |
| UNICODE_LEADERS_PAT | = | /\A(#{codepoints_to_pattern(UNICODE_LEADERS_AND_TRAILERS)})+/ |
| UCD | = | UnicodeDatabase.new |
| UniCode Database | ||
Returns a copy of str with the first character converted to uppercase and the remainder to lowercase
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 193
193: def capitalize(str)
194: upcase(slice(str, 0..0)) + downcase(slice(str, 1..-1) || '')
195: end
Perform composition on the characters in the string
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 229
229: def compose(str)
230: compose_codepoints u_unpack(str).pack('U*')
231: end
Checks if the string is valid UTF8.
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 259
259: def consumes?(str)
260: # Unpack is a little bit faster than regular expressions
261: begin
262: str.unpack('U*')
263: true
264: rescue ArgumentError
265: false
266: end
267: end
Perform decomposition on the characters in the string
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 224
224: def decompose(str)
225: decompose_codepoints(:canonical, u_unpack(str)).pack('U*')
226: end
Convert characters in the string to lowercase
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 190
190: def downcase(str); to_case :lowercase_mapping, str; end
Returns the number of grapheme clusters in the string. This method is very likely to be moved or renamed in future versions.
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 271
271: def g_length(str)
272: g_unpack(str).length
273: end
Returns the position of the passed argument in the string, counting in codepoints
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 139
139: def index(str, *args)
140: bidx = str.index(*args)
141: bidx ? (u_unpack(str.slice(0...bidx)).size) : nil
142: end
Inserts the passed string at specified codepoint offsets
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 129
129: def insert(str, offset, fragment)
130: str.replace(
131: u_unpack(str).insert(
132: offset,
133: u_unpack(fragment)
134: ).flatten.pack('U*')
135: )
136: end
Does Unicode-aware lstrip
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 150
150: def lstrip(str)
151: str.gsub(UNICODE_LEADERS_PAT, '')
152: end
Returns the KC normalization of the string by default. NFKC is considered the best normalization form for passing strings to databases and validations.
- str: The string to perform normalization on.
- form: The form you want to normalize in. Should be one of the following: :c, :kc, :d or :kd.
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 206
206: def normalize(str, form=ActiveSupport::Multibyte::DEFAULT_NORMALIZATION_FORM)
207: # See http://www.unicode.org/reports/tr15, Table 1
208: codepoints = u_unpack(str)
209: case form
210: when :d
211: reorder_characters(decompose_codepoints(:canonical, codepoints))
212: when :c
213: compose_codepoints reorder_characters(decompose_codepoints(:canonical, codepoints))
214: when :kd
215: reorder_characters(decompose_codepoints(:compatability, codepoints))
216: when :kc
217: compose_codepoints reorder_characters(decompose_codepoints(:compatability, codepoints))
218: else
219: raise ArgumentError, "#{form} is not a valid normalization variant", caller
220: end.pack('U*')
221: end
Reverses codepoints in the string.
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 166
166: def reverse(str)
167: u_unpack(str).reverse.pack('U*')
168: end
Does Unicode-aware rstrip
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 145
145: def rstrip(str)
146: str.gsub(UNICODE_TRAILERS_PAT, '')
147: end
Returns the number of codepoints in the string
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 160
160: def size(str)
161: u_unpack(str).size
162: end
Implements Unicode-aware slice with codepoints. Slicing on one point returns the codepoints for that character.
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 172
172: def slice(str, *args)
173: if (args.size == 2 && args.first.is_a?(Range))
174: raise TypeError, 'cannot convert Range into Integer' # Do as if we were native
175: elsif args[0].kind_of? Range
176: cps = u_unpack(str).slice(*args)
177: cps.nil? ? nil : cps.pack('U*')
178: elsif args.size == 1 && args[0].kind_of?(Numeric)
179: u_unpack(str)[args[0]]
180: else
181: u_unpack(str).slice(*args).pack('U*')
182: end
183: end
Removed leading and trailing whitespace
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 155
155: def strip(str)
156: str.gsub(UNICODE_LEADERS_PAT, '').gsub(UNICODE_TRAILERS_PAT, '')
157: end
Replaces all the non-utf-8 bytes by their iso-8859-1 or cp1252 equivalent resulting in a valid utf-8 string
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 276
276: def tidy_bytes(str)
277: str.split(//u).map do |c|
278: if !UTF8_PAT.match(c)
279: n = c.unpack('C')[0]
280: n < 128 ? n.chr :
281: n < 160 ? [UCD.cp1252[n] || n].pack('U') :
282: n < 192 ? "\xC2" + n.chr : "\xC3" + (n-64).chr
283: else
284: c
285: end
286: end.join
287: end
Used to translate an offset from bytes to characters, for instance one received from a regular expression match
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 238
238: def translate_offset(str, byte_offset)
239: return 0 if str == ''
240: return nil if byte_offset.nil?
241: chunk = str[0..byte_offset]
242: begin
243: begin
244: chunk.unpack('U*').length - 1
245: rescue ArgumentError => e
246: chunk = str[0..(byte_offset+=1)]
247: # Stop retrying at the end of the string
248: raise e unless byte_offset < chunk.length
249: # We damaged a character, retry
250: retry
251: end
252: # Catch the ArgumentError so we can throw our own
253: rescue ArgumentError
254: raise EncodingError.new('malformed UTF-8 character')
255: end
256: end
Convert characters in the string to uppercase
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 187
187: def upcase(str); to_case :uppercase_mapping, str; end
Compose decomposed characters to the composed form
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 393
393: def compose_codepoints(codepoints)
394: pos = 0
395: eoa = codepoints.length - 1
396: starter_pos = 0
397: starter_char = codepoints[0]
398: previous_combining_class = -1
399: while pos < eoa
400: pos += 1
401: lindex = starter_char - HANGUL_LBASE
402: # -- Hangul
403: if 0 <= lindex and lindex < HANGUL_LCOUNT
404: vindex = codepoints[starter_pos+1] - HANGUL_VBASE rescue vindex = -1
405: if 0 <= vindex and vindex < HANGUL_VCOUNT
406: tindex = codepoints[starter_pos+2] - HANGUL_TBASE rescue tindex = -1
407: if 0 <= tindex and tindex < HANGUL_TCOUNT
408: j = starter_pos + 2
409: eoa -= 2
410: else
411: tindex = 0
412: j = starter_pos + 1
413: eoa -= 1
414: end
415: codepoints[starter_pos..j] = (lindex * HANGUL_VCOUNT + vindex) * HANGUL_TCOUNT + tindex + HANGUL_SBASE
416: end
417: starter_pos += 1
418: starter_char = codepoints[starter_pos]
419: # -- Other characters
420: else
421: current_char = codepoints[pos]
422: current = UCD[current_char]
423: if current.combining_class > previous_combining_class
424: if ref = UCD.composition_map[starter_char]
425: composition = ref[current_char]
426: else
427: composition = nil
428: end
429: unless composition.nil?
430: codepoints[starter_pos] = composition
431: starter_char = composition
432: codepoints.delete_at pos
433: eoa -= 1
434: pos -= 1
435: previous_combining_class = -1
436: else
437: previous_combining_class = current.combining_class
438: end
439: else
440: previous_combining_class = current.combining_class
441: end
442: if current.combining_class == 0
443: starter_pos = pos
444: starter_char = codepoints[pos]
445: end
446: end
447: end
448: codepoints
449: end
Decompose composed characters to the decomposed form
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 372
372: def decompose_codepoints(type, codepoints)
373: codepoints.inject([]) do |decomposed, cp|
374: # if it's a hangul syllable starter character
375: if HANGUL_SBASE <= cp and cp < HANGUL_SLAST
376: sindex = cp - HANGUL_SBASE
377: ncp = [] # new codepoints
378: ncp << HANGUL_LBASE + sindex / HANGUL_NCOUNT
379: ncp << HANGUL_VBASE + (sindex % HANGUL_NCOUNT) / HANGUL_TCOUNT
380: tindex = sindex % HANGUL_TCOUNT
381: ncp << (HANGUL_TBASE + tindex) unless tindex == 0
382: decomposed.concat ncp
383: # if the codepoint is decomposable in with the current decomposition type
384: elsif (ncp = UCD[cp].decomp_mapping) and (!UCD[cp].decomp_type || type == :compatability)
385: decomposed.concat decompose_codepoints(type, ncp.dup)
386: else
387: decomposed << cp
388: end
389: end
390: end
Reverse operation of g_unpack
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 338
338: def g_pack(unpacked)
339: unpacked.flatten
340: end
Unpack the string at grapheme boundaries instead of codepoint boundaries
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 307
307: def g_unpack(str)
308: codepoints = u_unpack(str)
309: unpacked = []
310: pos = 0
311: marker = 0
312: eoc = codepoints.length
313: while(pos < eoc)
314: pos += 1
315: previous = codepoints[pos-1]
316: current = codepoints[pos]
317: if (
318: # CR X LF
319: one = ( previous == UCD.boundary[:cr] and current == UCD.boundary[:lf] ) or
320: # L X (L|V|LV|LVT)
321: two = ( UCD.boundary[:l] === previous and in_char_class?(current, [:l,:v,:lv,:lvt]) ) or
322: # (LV|V) X (V|T)
323: three = ( in_char_class?(previous, [:lv,:v]) and in_char_class?(current, [:v,:t]) ) or
324: # (LVT|T) X (T)
325: four = ( in_char_class?(previous, [:lvt,:t]) and UCD.boundary[:t] === current ) or
326: # X Extend
327: five = (UCD.boundary[:extend] === current)
328: )
329: else
330: unpacked << codepoints[marker..pos-1]
331: marker = pos
332: end
333: end
334: unpacked
335: end
Detect whether the codepoint is in a certain character class. Primarily used by the grapheme cluster support.
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 293
293: def in_char_class?(codepoint, classes)
294: classes.detect { |c| UCD.boundary[c] === codepoint } ? true : false
295: end
Re-order codepoints so the string becomes canonical
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 356
356: def reorder_characters(codepoints)
357: length = codepoints.length- 1
358: pos = 0
359: while pos < length do
360: cp1, cp2 = UCD[codepoints[pos]], UCD[codepoints[pos+1]]
361: if (cp1.combining_class > cp2.combining_class) && (cp2.combining_class > 0)
362: codepoints[pos..pos+1] = cp2.code, cp1.code
363: pos += (pos > 0 ? -1 : 1)
364: else
365: pos += 1
366: end
367: end
368: codepoints
369: end
Convert characters to a different case
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 343
343: def to_case(way, str)
344: u_unpack(str).map do |codepoint|
345: cp = UCD[codepoint]
346: unless cp.nil?
347: ncp = cp.send(way)
348: ncp > 0 ? ncp : codepoint
349: else
350: codepoint
351: end
352: end.pack('U*')
353: end
Unpack the string at codepoints boundaries
[ show source ]
# File vendor/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb, line 298
298: def u_unpack(str)
299: begin
300: str.unpack 'U*'
301: rescue ArgumentError
302: raise EncodingError.new('malformed UTF-8 character')
303: end
304: end