10:52 a.m.
For the Wine project we are trying to make it as easy as possible for
Wine contributors to modify and rebuild the Wine documentation. The Po4a
dependencies have caused some concern in this regard: for each
dependency the contributor will have to track down which package to
install for his distribution and this makes the initial setup
significantly harder.
Po4a being not very widespread we have checked it in the Winedocs CVS
(http://cvs.sourceforge.net/viewcvs.py/wine/docs/). But Po4a has quite a
few dependencies so this does not help very much. So now we are hoping
to make it possible to reduce the number of po4a dependencies. Here is
the list I came up with, together with some notes for each of them:
* Locale::gettext (perl module)
Needed by po4a for localization.
Provided by liblocale-gettext-perl on Debian, perl-Locale-gettext on
Mandrake and Fedora Core(DAG), perl-gettext on SUSE.
Would you be open to a patch that acted as a wrapper around
Locale::gettext so that po4a would continue to work untranslated if that
module was missing?
The idea we would add a 'Po4aGettext' module that would export
gettext() and dgettext() functions. These would try to load
Locale::gettext in an eval function and use it if that succeeds.
Otherwise the Po4aGettext would simply return the untranslated strings.
The only changes to the other Po4a modules would be replacing 'use
Locale::gettext' with 'use Po4aGettext'.
* Text::WrapI18N (perl module)
Pure perl (so easy to check in) but depends on Text::CharWidth which
is not pure perl.
Provided by libtext-wrapi18n-perl on Debian. Found no RPM packages
providing it.
Text::WrapI18N was not used in po4a 0.16.2. I initially thought it
was used to wrap the text being output to the .po and .sgml files but in
fact it seems to only be used to print messages, warnings and errors.
Why is it needed? Doesn't a simple print work fine?
* Term::ReadKey (perl module)
Provided by libterm-readkey-perl on Debian, perl-Term-ReadKey on
Mandrake and Fedora Core(DAG), perl-TermReadKey on SUSE.
This module is used to determine the size of the terminal. This
information is then used by the wrap functions.
* SGMLS (perl module)
Needed by po4a to interface with the nsgmls parser.
Pure perl (so easy to check in).
Provided by libsgmls-perl on Debian, perl-SGMLSpm on Mandrake and
Fedora Core, perl-SGMLS on SUSE.
This is an essential part of po4a. The easiest way to remove this
dependency would be to check it into the Winedocs repository. So no
action needed on the Po4a side.
* nsgmls (command line tool)
Needed by po4a to parse the Sgml files.
Provided by sp on Debian, sgml-tools or openjade on Mandrake and
Fedora Core, opensp on SUSE.
This is an essential part of po4a. So it will have to remain as a
dependency which is reasonable.
--
Francois Gouget
fgouget(a)codeweavers.com
12:49 p.m.
On Thu, May 26, 2005 at 05:52:39PM +0200, Francois Gouget wrote:
For the Wine project we are trying to make it as easy as possible for
Wine contributors to modify and rebuild the Wine documentation. The Po4a
dependencies have caused some concern in this regard: for each
dependency the contributor will have to track down which package to
install for his distribution and this makes the initial setup
significantly harder.
Po4a being not very widespread we have checked it in the Winedocs CVS
(http://cvs.sourceforge.net/viewcvs.py/wine/docs/). But Po4a has quite a
few dependencies so this does not help very much. So now we are hoping
to make it possible to reduce the number of po4a dependencies. Here is
the list I came up with, together with some notes for each of them:
* Locale::gettext (perl module)
Needed by po4a for localization.
Provided by liblocale-gettext-perl on Debian, perl-Locale-gettext on
Mandrake and Fedora Core(DAG), perl-gettext on SUSE.
Would you be open to a patch that acted as a wrapper around
Locale::gettext so that po4a would continue to work untranslated if that
module was missing?
The idea we would add a 'Po4aGettext' module that would export
gettext() and dgettext() functions. These would try to load
Locale::gettext in an eval function and use it if that succeeds.
Otherwise the Po4aGettext would simply return the untranslated strings.
The only changes to the other Po4a modules would be replacing 'use
Locale::gettext' with 'use Po4aGettext'.
Ok. Great. It could even be done by default in the Common.pm module, which
would then export the d?gettext functions. Impact on other parts would be
the need to kill the explicit gettext loading.
* Text::WrapI18N (perl module)
Pure perl (so easy to check in) but depends on Text::CharWidth which
is not pure perl.
Provided by libtext-wrapi18n-perl on Debian. Found no RPM packages
providing it.
Text::WrapI18N was not used in po4a 0.16.2. I initially thought it
was used to wrap the text being output to the .po and .sgml files but in
fact it seems to only be used to print messages, warnings and errors.
Why is it needed? Doesn't a simple print work fine?
This module becomes important when you want to wrap CKJ languages
(japaneese, corean), which don't have any spaces, if I understood well. So
finding where you can cut the sentence properly is not as easy as in, say,
french.
So, actually, we *ought* to use it all around the place. Maybe through a
wrapper such as the one you propose for gettext...
* Term::ReadKey (perl module)
Provided by libterm-readkey-perl on Debian, perl-Term-ReadKey on
Mandrake and Fedora Core(DAG), perl-TermReadKey on SUSE.
This module is used to determine the size of the terminal. This
information is then used by the wrap functions.
But if it's not there, things should still work, using 80 as terminal size.
* SGMLS (perl module)
Needed by po4a to interface with the nsgmls parser.
Pure perl (so easy to check in).
Provided by libsgmls-perl on Debian, perl-SGMLSpm on Mandrake and
Fedora Core, perl-SGMLS on SUSE.
This is an essential part of po4a. The easiest way to remove this
dependency would be to check it into the Winedocs repository. So no
action needed on the Po4a side.
* nsgmls (command line tool)
Needed by po4a to parse the Sgml files.
Provided by sp on Debian, sgml-tools or openjade on Mandrake and
Fedora Core, opensp on SUSE.
This is an essential part of po4a. So it will have to remain as a
dependency which is reasonable.
Actually, I'd like since a long time to kill this one, changing it to an
home-made parser such as the one used in xml or so...
Bye, Mt.
4:40 p.m.
On Thu, May 26, 2005 at 07:49:44PM +0200, Martin Quinson wrote:
On Thu, May 26, 2005 at 05:52:39PM +0200, Francois Gouget wrote:
[...]
> * Text::WrapI18N (perl module)
> Pure perl (so easy to check in) but depends on Text::CharWidth which
> is not pure perl.
> Provided by libtext-wrapi18n-perl on Debian. Found no RPM packages
> providing it.
>
> Text::WrapI18N was not used in po4a 0.16.2. I initially thought it
> was used to wrap the text being output to the .po and .sgml files but in
> fact it seems to only be used to print messages, warnings and errors.
> Why is it needed? Doesn't a simple print work fine?
This module becomes important when you want to wrap CKJ languages
(japaneese, corean), which don't have any spaces, if I understood well. So
finding where you can cut the sentence properly is not as easy as in, say,
french.
So, actually, we *ought* to use it all around the place. Maybe through a
wrapper such as the one you propose for gettext...
AFAICT this module is only useful to print error messages, so this is
all about cosmetics and a wrapper (no, this one does not wrap lines ;))
should be fine. When this module is not present, no wrapping is performed.
Jordi, is it fine by you?
[...]
> * nsgmls (command line tool)
> Needed by po4a to parse the Sgml files.
> Provided by sp on Debian, sgml-tools or openjade on Mandrake and
> Fedora Core, opensp on SUSE.
>
> This is an essential part of po4a. So it will have to remain as a
> dependency which is reasonable.
Actually, I'd like since a long time to kill this one, changing it to an
home-made parser such as the one used in xml or so...
I thought you had no time ;)
Denis
2:44 a.m.
On Thu, May 26, 2005 at 11:40:27PM +0200, Denis Barbier wrote:
On Thu, May 26, 2005 at 07:49:44PM +0200, Martin Quinson wrote:
> On Thu, May 26, 2005 at 05:52:39PM +0200, Francois Gouget wrote:
[...]
> > * Text::WrapI18N (perl module)
> > Pure perl (so easy to check in) but depends on Text::CharWidth which
> > is not pure perl.
> > Provided by libtext-wrapi18n-perl on Debian. Found no RPM packages
> > providing it.
> >
> > Text::WrapI18N was not used in po4a 0.16.2. I initially thought it
> > was used to wrap the text being output to the .po and .sgml files but in
> > fact it seems to only be used to print messages, warnings and errors.
> > Why is it needed? Doesn't a simple print work fine?
>
> This module becomes important when you want to wrap CKJ languages
> (japaneese, corean), which don't have any spaces, if I understood well. So
> finding where you can cut the sentence properly is not as easy as in, say,
> french.
>
> So, actually, we *ought* to use it all around the place. Maybe through a
> wrapper such as the one you propose for gettext...
AFAICT this module is only useful to print error messages, so this is
all about cosmetics and a wrapper (no, this one does not wrap lines ;))
should be fine. When this module is not present, no wrapping is performed.
s/ought/should/
I know we don't use it very much, but I persist to say that it would be
better if we did so.
I now remember why it's not used in the Po.pm module (but there is no excuse
why it's not used in Sgml.pm and such). We have to preserve the space at the
end of the line here, and all wrapping modules keep eating it. They change:
"toto tutu toto tutu "
to
"toto tutu"
"toto tutu"
But to preserve the po file semantic, we need:
"toto tutu "
"toto tutu "
Explaining why we keep using a wrapping algorithm which is known to be
broken for some languages. I wanted for ages to fill a wishlist bug report
asking Text::WrapI18N to provide an "space preservation" option. But I
didn't yet, my bad. I'll try to do so soon.
Now, about using it through a wrapper, I think this is a good idea. The
wrapper could even be smart enough to detect that you're using thai (or
another language making this module mandatory) encoding (or chars, if UTF8),
you're going into problem and emit a warning.
> > * nsgmls (command line tool)
> > Needed by po4a to parse the Sgml files.
> > Provided by sp on Debian, sgml-tools or openjade on Mandrake and
> > Fedora Core, opensp on SUSE.
> >
> > This is an essential part of po4a. So it will have to remain as a
> > dependency which is reasonable.
>
> Actually, I'd like since a long time to kill this one, changing it to an
> home-made parser such as the one used in xml or so...
I thought you had no time ;)
Sure. Once again, I was unclear, sorry. I'd love to see disapear, but won't
do anything for this myself in the next months. I just wanted to note that
the only dependencies Francois were ok with are the ones I'd like the most
to kill :)
Bye, Mt.
3:45 a.m.
Martin Quinson wrote:
[...]
> * Locale::gettext (perl module)
> Needed by po4a for localization.
> Provided by liblocale-gettext-perl on Debian, perl-Locale-gettext on
>Mandrake and Fedora Core(DAG), perl-gettext on SUSE.
>
> Would you be open to a patch that acted as a wrapper around
>Locale::gettext so that po4a would continue to work untranslated if that
>module was missing?
[...]
Ok. Great. It could even be done by default in the Common.pm module,
which
would then export the d?gettext functions. Impact on other parts would be
the need to kill the explicit gettext loading.
Good idea. I'll do that.
> * Text::WrapI18N (perl module)
> Pure perl (so easy to check in) but depends on Text::CharWidth which
>is not pure perl.
> Provided by libtext-wrapi18n-perl on Debian. Found no RPM packages
>providing it.
>
> Text::WrapI18N was not used in po4a 0.16.2. I initially thought it
>was used to wrap the text being output to the .po and .sgml files but in
>fact it seems to only be used to print messages, warnings and errors.
>Why is it needed? Doesn't a simple print work fine?
This module becomes important when you want to wrap CKJ languages
(japaneese, corean), which don't have any spaces, if I understood well. So
finding where you can cut the sentence properly is not as easy as in, say,
french.
So, actually, we *ought* to use it all around the place. Maybe through a
wrapper such as the one you propose for gettext...
Ok. I can understand why it would be important to use it for writing po
files and Sgml files. It's a shame it's used everywhere but there ;-)
(I've seen your other email explaining why it's that way).
What I don't understand is why it is used to print informational and
error messages. Won't the xterm wrap things on its own? It does just
fine when I print a very long line in French or English. Doesn't it do
the same CKJ languages? If not I'd say it is pretty broken.
My next question has to do with the wrapper implementation. I did not
check it out in details but it seems like it works on the multibyte
string. Wouldn't it be possible to implement a wrapper using the Perl
5.8 Unicode support? It would work like this:
# "Noël Noël" in UTF-8
my $oct="Noël Noël";
Here length($oct) = 11 because ë is encoded as two 8bit characters.
my $str=Encode::decode("UTF-8", $oct);
Here length($str) = 9 because ë is encoded as one unicode
character. This means we can cut up the string as we want:
my $sub=Encode::encode("UTF-8",substr($str,0,3));
Here $sub contains "Noë", that is "Noë" in UTF-8 and has a
length
of 4.
There's still the issue of word boundaries because, at least for the
Sgml file, we would not want to cut a word in two. But I would expect
that, on Unicode strings, \b, \w amd \W would work sensibly even for CKJ
languages. The advantage is that this would make it possible to do
wrapping using only standard Perl features. The drawback is it would
required perl 5.8.
--
Francois Gouget
fgouget(a)codeweavers.com
4:14 a.m.
On Fri, May 27, 2005 at 10:45:42AM +0200, Francois Gouget wrote:
Martin Quinson wrote:
[...]
>>* Text::WrapI18N (perl module)
Ok. I can understand why it would be important to use it for writing po
files and Sgml files. It's a shame it's used everywhere but there ;-)
(I've seen your other email explaining why it's that way).
exactly.
What I don't understand is why it is used to print informational
and
error messages. Won't the xterm wrap things on its own? It does just
fine when I print a very long line in French or English. Doesn't it do
the same CKJ languages? If not I'd say it is pretty broken.
It may be seen as cosmetics. We wanted the error location to be more
obvious, something like:
toto.sgml:43: you stupid user, you did it again and gave me a file where the
<p> tag is not closed, even if I told you not later than 5
minutes ago that you shouldn't.
toto.sgml:46: Oh well. This file is really crappy: you don't even close the
<a> tag. This *is* an error you ought to fix.
It's not a plain wrapping, but respects the indentation we wanted to make it
look better (in our opinions). For sure, when the dependency is not there,
you could use print() without wrapping, it wouldn't breaking anything
(beside my/our sense of beauty wrt error messages ;).
My next question has to do with the wrapper implementation. I did not
check it out in details but it seems like it works on the multibyte
string. Wouldn't it be possible to implement a wrapper using the Perl
5.8 Unicode support?
Maybe. But this version is not really widespread yet, is it? If you come up
with a working implementation of that (I'm really bad at such issues and
leave Denis or Jordi answer), you could test whether we are using perl >=
5.8 and if not, fallback to libtext-wrap18n-perl, couldn't you?
You could also not wrap anything when the dependency isn't there. As temp
solution, it would perfectly work, I guess.
Mt.
6:45 p.m.
Martin Quinson wrote:
[...]
>My next question has to do with the wrapper implementation. I did
not
>check it out in details but it seems like it works on the multibyte
>string. Wouldn't it be possible to implement a wrapper using the Perl
>5.8 Unicode support?
Maybe. But this version is not really widespread yet, is it?
Actually it is quite widespread. It ships with Debian testing, RedHat >=
8.0, all Fedora Core obviously, SuSE >= 8.2, Mandrake >= 9.1 and maybe
even some older versions of these distributions.
But the point is moot since po4a already depends on the Encode module:
it's used in Man.pm, Po.pm, Text.pm and TransTractor.pm. I think this
means po4a already depends on perl 5.8 but maybe I'm wrong.
Anyway, the Encode module is the one that's needed for the
implementation I sketched.
If you come up
with a working implementation of that (I'm really bad at such issues and
leave Denis or Jordi answer), you could test whether we are using perl >=
5.8 and if not, fallback to libtext-wrap18n-perl, couldn't you?
Unfortunately I'm not sure I can come up with a working implementation.
I'd have to look at WrapI18N in more details first. I'm sure I'm missing
something.
Also the problem if I did come up with something is that I don't know
enough about CKJ languages to actually verify my implementation would
work with them.
You could also not wrap anything when the dependency isn't there.
As temp
solution, it would perfectly work, I guess.
Yes, that seems like a much easier target. I'll try that first <g>.
--
Francois Gouget
fgouget(a)codeweavers.com
8:20 a.m.
New subject: [Po4a-devel]Po4a dependency on libtext-wrapi18n-perl
Ok, I had another look on this one, and it looks to me that we could quite
easily drop the dependency. The code of this library itself is very close of
what I did in the wrapper part, I think.
The first difficulty is to wrap where it's legal, ie after space or after
every CJK char. For that, we could use the same trick than
libtext-wrapi18n-perl:
>>>>>>>>>>>>>>>>>
# Returns 1 for Chinese and Japanese characters. This means that
# these characters allow line wrapping after this character even
# without whitespaces because these languages don't use whitespaces
# between words.
#
# Character must be given in UCS-4 codepoint value.
sub _isCJ {
my $u=shift;
if ($u >= 0x3000 && $u <= 0x312f) {
if ($u == 0x300a || $u == 0x300c || $u == 0x300e ||
$u == 0x3010 || $u == 0x3014 || $u == 0x3016 ||
$u == 0x3018 || $u == 0x301a) {return 0;}
return 1;
} # CJK punctuations, Hiragana, Katakana, Bopomofo
if ($u >= 0x31a0 && $u <= 0x31bf) {return 1;} # Bopomofo
if ($u >= 0x31f0 && $u <= 0x31ff) {return 1;} # Katakana extension
if ($u >= 0x3400 && $u <= 0x9fff) {return 1;} # Han Ideogram
if ($u >= 0xf900 && $u <= 0xfaff) {return 1;} # Han Ideogram
if ($u >= 0x20000 && $u <= 0x2ffff) {return 1;} # Han Ideogram
return 0;
}
<<<<<<<<<<<<<<<<<<<<
Then, the second difficulty is to compute the char width in column right.
Google finds this:
http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
Reimplementing this in perl seems quite easy. But I'm not sure this source
can be trusted, and I feel just too lazy to download the libc source to see
how it's done in there...
Bye, Mt.
6895
days inactive
6903
days old
7 comments
3 participants
participants (3)
-
Denis Barbier -
Francois Gouget -
Martin Quinson