Server side text layout (was Re: [forum] some XFree86 5.0 questions...)

[Owen Taylor]

On Tue, 2003-03-25 at 03:37, Alexander Gelfenbain wrote:
> On Sun, Mar 23, 2003 at 01:09:03AM -0500, Owen Taylor wrote:
> > > That is true only within assumptions of the current text rendering
> > > model. When applications use an encapsulated API like STSF, they
> > > do not require glyph metric data. 
> > > 
> > > The advantage of encapsulated APIs is that the only thing applications 
> > > need to do is to create a single object that represent a fragment of 
> > > displayable text.
> > > 
> > > Such an object:
> > > 	
> > > 	- manages its highlighted regions 
> > > 	
> > > 	- can position a caret without external intervention
> > > 	
> > > 	- can be rendered in an atomic operation
> > > 
> > > Different encapsulated APIs tend to call such an object differently:
> > > 
> > > 	STSF calls it a "Line Object."
> > > 	ATSUI and Java2D calls it a "Text Layout Object." 
> > > 
> > > Once such an object is created, the application does not need to know
> > > any low-level details about text that it used to create it, including
> > > metric data, font technology, etc. 
> > 
> > There is a very big leap of faith here from the idea that it's
> > good to have an encapsulated paragraph object, to the idea that
> > it's good to put it on an out-of-process server.
> 
> I was saying that encapsulated APIs remove the need to transmit
> individual character metrics over the wire; and I don't think 
> you are disagreeing with me.

I'm not disagreeing with you that it removes the need; I'm just
disagreeing with you that it is an appropriate method for
removing the need.

> Encapsulated paragraph objects are good in general, and they
> can be implemented either client-side or server-side. For example
> STSF has two modes of operation - server side with STSF-enabled
> X11 server and client-side with X11 STSF client library. 

This is perhaps where we part company ... you seem to think
it is a good thing that STSF allows either client side or 
server side layout. I think it is a disaster to have that
sort of "flexible architecture". 

What I'd predict would happen is that most people will use 
and develop with the client side libs, and then when someone 
tries to use them with the server side layout, performance 
will be abysmal, because you can get away with a lot of
stuff with the client side libs you can't with remote layout.

So, the server-side layout capabilities become just a bullet-point.
They are there, but no app works well with them. It's sort
of like the internationalization situation with the traditional
POSIX API's. Sure, you get wonderful encoding independence
if you use all the appropriate mb* functions, but most programmers
just assumed byte==character, and people who cared about i18n had 
to go back and fix up every program.

> Server-side implementation is potentially easier on resources 
> (everything can be shared among multiple clients and even X servers)
> and on the protocol (objects are referenced by integer handles.)
>
> > Possibly objections include:
> > 
> >  - Layout latency matters even if you can do it on a paragraph
> >    level. A text editor needs to find the layed-out size of tens of 
> >    thousands of paragraphs a second. 
> 
> That's very true for every client-server architecture, but it can
> be remedied by adding requests that do operations on arrays
> of data in one protocol roundtrip. 

We get a fair amount of resistance in GTK+ to application
authors needing to use layout objects at all rather than simple
XTextWidth() and XDrawText() style calls. Adding the need to chunk 
up operations ... ouch. Even doing this for, say, GtkTextView,
would be hard: for expose event during incremental reflow, 
GtkTextView, for instance, lays out paragraphs until it has enough 
to cover the exposed area.

But even worse, would be trying to avoid lots of round-trips
during widget layout. Imagine a  menu with a bunch of menu items,
each with a label whose size needs to be determined. Since you
presumably don't want your menu to gradually increase in size as
responses come back from the server, you'd have to add some sort 
of "I have my size now" flag to the GTK+ geometry system. (*)

> A text editor needs to find the layed-out size of a lot of paragraphs
> when it loads the document. During the editing process, however,
> the most typical client-server operation would be client telling server
> "text changed from offset X to offset Y, the new text is Z, 
> recalculate the size of the paragpraph and redisplay it and remaining
> visible paragraphs"  Only one roundtrip.

Well,editing with client side layout objects requires _0_ roundtrips 
per keystroke. 

> >  - The paragraph object is seldom going to be simple. PangoLayout
> >    (Pango's paragraph object) has 40-50 methods, the Java class
> >    seems to be similar.
> 
> We have 76 methods for Style/Text/Line objects in STSF now. Not
> all of them are unique protocol requests though.
> 
> > 
> >    Putting this degree of complexity at a protocol layer seems
> >    a little bit questionable to me.
> 
> Why?

I guess it depends on whether you consider the protocol to
be something exposed and stabilized or just an implementation
detail. If it's just a implementation detail it doesn't 
matter much, but if you want protocol stability, well, if there
are 76 methods today, it doesn't seem unlikely that you'll
need 77 tomorrow.

> >  - There may be needs to customize the layout process; to wrap
> >    paragraphs to custom shapes, to implement custom line break
> >    algorithms (say, to do global paragraph optimization) and
> >    so forth. The Java2D way of doing this customization seems
> >    to involve creating TextLayout objects for smaller pieces of 
> >    text and positioning them youself:
> > 
> > http://java.sun.com/j2se/1.4.1/docs/api/java/awt/font/LineBreakMeasurer.html
> > 
> >    Which certainly won't work well if there is significant latency
> >    for creating text-layout objects and finding out their
> >    size.
> 
> 
> STSF has special objects - STGlyphVector objects - that encapsulate
> arrays of positioned and styled glyphs. If an application chose to 
> use STGlyphVector objects, it indeed needs to query all metric data
> for characters and position them. But after that the entire 
> STGlyphVector object (which can be multi-line or multi-paragraph)
> is displayable in one protocol request (that does not even require
> a roundrtip.)

So, the choice is between:

 - Get good performance and i18n, but be restricted to exactly the
   layout that STSF provides.

 - Get fully flexible layout, but get poor performance (the metric  
   set is typically larger than the glyph images that need to be 
   rendered), have to reimplement all of layout yourself, including  
   complexities like bidi, and lose most of the i18n capabilities (I
   assume there is no representation of, say, OpenType layout tables
   in the the STSF get-the-metrics API call.)

?

Regards,
                                           Owen

(*) To be completely upfront, I'm to some extent arguing against
    a straw man here. A round-trip-per-item for a menu with 50 items
    only becomes a problem when the layout server is network
    remote. Presumably, nobody would use the configuration:

     app <==network==> layout server <=> X server

   But rather:

     app <=> layout server <==network==> X server

   But I do think that the text editor case is a problem; I do
   think the general increase in application API complexity
   for achieving optimal performance is a big problem. And
   I don't want the idea that encapsulated layout objects 
   fixes the roundtrip and metrics problems to stand.