1107 lines
60 KiB
Plaintext
1107 lines
60 KiB
Plaintext
|
|
JPEG SYSTEM ARCHITECTURE 3-OCT-91
|
|
|
|
|
|
This file provides an overview of the "architecture" of the portable JPEG
|
|
software; that is, the functions of the various modules in the system and the
|
|
interfaces between modules. For more precise details about any data structure
|
|
or calling convention, see the header files.
|
|
|
|
Important note: when I say "module" I don't mean "a C function", which is what
|
|
some people seem to think the term means. A separate C source file is closer
|
|
to the mark. Also, it is frequently the case that several different modules
|
|
present a common interface to callers; the term "object" or "method" refers to
|
|
this common interface (see "Poor man's object-oriented programming", below).
|
|
|
|
JPEG-specific terminology follows the JPEG R9 draft:
|
|
A "component" means a color channel, e.g., Red or Luminance.
|
|
A "sample" is a pixel component value (i.e., one number in the image data).
|
|
A "coefficient" is a frequency coefficient (a DCT transform output number).
|
|
The term "block" refers to an 8x8 group of samples or coefficients.
|
|
"MCU" (minimum coded unit) is the same as "MDU" of the R8 draft; i.e., an
|
|
interleaved set of blocks of size determined by the sampling factors,
|
|
or a single block in a noninterleaved scan.
|
|
|
|
|
|
*** System requirements ***
|
|
|
|
We must support compression and decompression of both Huffman and
|
|
arithmetic-coded JPEG files. Any set of compression parameters allowed by the
|
|
JPEG spec should be readable for decompression. (We can be more restrictive
|
|
about what formats we can generate.) (Note: for legal reasons no arithmetic
|
|
coding implementation is currently included in the publicly available sources.
|
|
However, the architecture still supports it.)
|
|
|
|
We need to be able to handle both raw JPEG files (more specifically, the JFIF
|
|
format) and JPEG-in-TIFF (C-cubed's format, and perhaps Kodak's). Even if we
|
|
don't implement TIFF ourselves, other people will want to use our code for
|
|
that. This means that generation and scanning of the file header has to be
|
|
separated out.
|
|
|
|
Perhaps we should be prepared to support the JPEG lossless mode (also referred
|
|
to in the spec as spatial DPCM coding). A lot of people seem to believe they
|
|
need this... whether they really do is debatable, but the customer is always
|
|
right. On the other hand, there will not be much sharable code between the
|
|
lossless and lossy modes! At best, a lossless program could be derived from
|
|
parts of the lossy version. For now we will only worry about the lossy mode.
|
|
|
|
I see no real value in supporting the JPEG progressive modes (note that
|
|
spectral selection and successive approximation are two different progressive
|
|
modes). These are only of interest when painting the decompressed image in
|
|
real-time, which nobody is going to do with a pure software implementation.
|
|
|
|
There is some value in supporting the hierarchical mode, which allows for
|
|
successive frames of higher resolution. This could be of use for including
|
|
"thumbnail" representations. Also, Storm's JPEG++ files probably use the
|
|
hierarchical mode (I haven't looked). However, this appears to add a lot more
|
|
complexity than it is worth.
|
|
|
|
A variety of uncompressed image file formats and user interfaces must be
|
|
supported. These aspects therefore have to be kept separate from the rest of
|
|
the system. A particularly important issue is whether color quantization of
|
|
the output is needed (i.e., whether a colormap is used). We should be able to
|
|
support both adaptive quantization (which requires two or more passes over the
|
|
image) and nonadaptive (quantization to a prespecified colormap, which can be
|
|
done in one pass).
|
|
|
|
Memory usage is an important concern, since we will port this code to 80x86
|
|
and other limited-memory machines. For large intermediate structures, we
|
|
should be able to use either virtual memory or temporary files.
|
|
|
|
It should be possible to build programs that handle compression only,
|
|
decompression only, or both, without much duplicate or unused code in any
|
|
version. (In particular, a decompression-only version should have no extra
|
|
baggage.)
|
|
|
|
|
|
*** Compression overview ***
|
|
|
|
The *logical* steps needed in (non-lossless) JPEG compression are:
|
|
|
|
1. Conversion from incoming image format to a standardized internal form
|
|
(either RGB or greyscale).
|
|
|
|
2. Color space conversion (e.g., RGB to YCbCr). This is a null step for
|
|
greyscale (unless we support mapping color inputs to greyscale, which
|
|
would most easily be done here). Gamma adjustment may also be needed here.
|
|
|
|
3. Subsampling (reduction of number of samples in some color components).
|
|
This step operates independently on each color component.
|
|
|
|
4. MCU extraction (creation of a single sequence of 8x8 sample blocks).
|
|
This step and the following ones are performed once for each scan
|
|
in the output JPEG file, i.e., once if making an interleaved file and more
|
|
than once for a noninterleaved file.
|
|
Note: both this step and the previous one must deal with edge conditions
|
|
for pictures that aren't a multiple of the MCU dimensions. Alternately,
|
|
we could expand the picture to a multiple of an MCU before doing these
|
|
two steps. (The latter seems better and has been adopted below.)
|
|
|
|
5. DCT transformation of each 8x8 block.
|
|
|
|
6. Quantization scaling and zigzag reordering of the elements in each 8x8
|
|
block.
|
|
|
|
7. Huffman or arithmetic encoding of the transformed block sequence.
|
|
|
|
8. Output of the JPEG file with whatever headers/markers are wanted.
|
|
|
|
Of course, the actual implementation will combine some of these logical steps
|
|
for efficiency. The trick is to keep these logical functions as separate as
|
|
possible without losing too much performance.
|
|
|
|
In addition to these logical pipeline steps, we need various modules that
|
|
aren't part of the data pipeline. These are:
|
|
|
|
A. Overall control (sequencing of other steps & management of data passing).
|
|
|
|
B. User interface; this will determine the input and output files, and supply
|
|
values for some compression parameters. Note that this module is highly
|
|
platform-dependent.
|
|
|
|
C. Compression parameter selection: some parameters should be chosen
|
|
automatically rather than requiring the user to find a good value.
|
|
The prototype only does this for the back-end (Huffman or arithmetic)
|
|
parameters, but further in the future, more might be done. A
|
|
straightforward approach to selection is to try several values; this
|
|
requires being able to repeatedly apply some portion of the pipeline and
|
|
inspect the results (without actually outputting them). Probably only
|
|
entropy encoding parameters can reasonably be done this way; optimizing
|
|
earlier steps would require too much data to be reprocessed (not to mention
|
|
the problem of interactions between parameters for different steps).
|
|
What other facilities do we need to support automatic parameter selection?
|
|
|
|
D. A memory management module to deal with small-memory machines. This must
|
|
create the illusion of virtual memory for certain large data structures
|
|
(e.g., the subsampled image or the transformed coefficients).
|
|
The interface to this must be defined to minimize the overhead incurred,
|
|
especially on virtual-memory machines where the module won't do much.
|
|
|
|
In many cases we can arrange things so that a data stream is produced in
|
|
segments by one module and consumed by another without the need to hold it all
|
|
in (virtual) memory. This is obviously not possible for any data that must be
|
|
scanned more than once, so it won't work everywhere.
|
|
|
|
The major variable at this level of detail is whether the JPEG file is to be
|
|
interleaved or not; that affects the order of processing so fundamentally that
|
|
the central control module must know about it. Some of the other modules may
|
|
need to know it too. It would simplify life if we didn't need to support
|
|
noninterleaved images, but that is not reasonable.
|
|
|
|
Many of these steps operate independently on each color component; the
|
|
knowledge of how many components there are, and how they are interleaved,
|
|
ought to be confined to the central control module. (Color space conversion
|
|
and MCU extraction probably have to know it too.)
|
|
|
|
|
|
*** Decompression overview ***
|
|
|
|
Decompression is roughly the inverse process from compression, but there are
|
|
some additional steps needed to produce a good output image.
|
|
|
|
The *logical* steps needed in (non-lossless) JPEG decompression are:
|
|
|
|
1. Scanning of the JPEG file, decoding of headers/markers etc.
|
|
|
|
2. Huffman or arithmetic decoding of the coefficient sequence.
|
|
|
|
3. Quantization descaling and zigzag reordering of the elements in each 8x8
|
|
block.
|
|
|
|
4. MCU disassembly (conversion of a possibly interleaved sequence of 8x8
|
|
blocks back to separate components in pixel map order).
|
|
|
|
5. (Optional) Cross-block smoothing per JPEG section 13.10 or a similar
|
|
algorithm. (Steps 5-8 operate independently on each component.)
|
|
|
|
6. Inverse DCT transformation of each 8x8 block.
|
|
|
|
7. De-subsampling. At this point a pixel image of the original dimensions
|
|
has been recreated.
|
|
|
|
8. Post-subsampling smoothing. This can be combined with de-subsampling,
|
|
by using a convolution-like calculation to generate each output pixel
|
|
directly from one or more input pixels.
|
|
|
|
9. Cropping to the original pixel dimensions (throwing away duplicated
|
|
pixels at the edges). It is most convenient to do this now, as the
|
|
preceding steps are simplified by not having to worry about odd picture
|
|
sizes.
|
|
|
|
10. Color space reconversion (e.g., YCbCr to RGB). This is a null step for
|
|
greyscale. (Note that if we support mapping color JPEG to greyscale,
|
|
it could be done as part of this step.) Gamma adjustment may also be
|
|
needed here.
|
|
|
|
11. Color quantization (only if a colormapped output format is requested).
|
|
NOTE: it might be better to do this on the internal color space instead of
|
|
RGB? If so, it would need to be performed one step earlier.
|
|
|
|
12. Writing of the desired image format.
|
|
|
|
As before, some of these will be combined into single steps. When dealing
|
|
with a noninterleaved JPEG file, steps 2-9 will be performed once for each
|
|
scan; the resulting data will need to be buffered up so that step 10 can
|
|
process all the color components together.
|
|
|
|
The same auxiliary modules are needed as before, except for compression
|
|
parameter selection. Note that rerunning a pipeline stage should never be
|
|
needed during decompression. This may allow a simpler control module. The
|
|
user interface might also be simpler since it need not supply any compression
|
|
parameters.
|
|
|
|
As before, not all of these steps require the whole image to be stored.
|
|
Actually, two-pass color quantization is the only step that logically requires
|
|
this; everything else could be done a few raster lines at a time (at least for
|
|
interleaved images). We might want to make color quantization be a separate
|
|
program because of this fact.
|
|
|
|
Again, many of the steps should be able to work on one color component in
|
|
ignorance of the other components.
|
|
|
|
|
|
*** Implications of noninterleaved formats ***
|
|
|
|
Much of the work can be done in a single pass if an interleaved JPEG file
|
|
format is used. With a noninterleaved JPEG file, separating or recombining
|
|
the components will force use of virtual memory (on a small-memory machine,
|
|
we probably would want one temp file per color component).
|
|
|
|
If any of the image formats we read or write are noninterleaved, the opposite
|
|
condition might apply: processing a noninterleaved JPEG file would be more
|
|
efficient. Offhand, though, I can't think of any popular image formats that
|
|
work that way; besides the win would only come if the same color space were
|
|
used in JPEG and non-JPEG files. It's not worth the complexity to make the
|
|
system design accommodate that case efficiently.
|
|
|
|
An argument against interleaving is that it makes the decompressor need more
|
|
memory for cross-block smoothing (since the minimum processable chunk of the
|
|
image gets bigger). With images more than 1000 pixels across, 80x86 machines
|
|
are likely to have difficulty in handling this feature.
|
|
|
|
Another argument against interleaving is that the noninterleaved format allows
|
|
a wider range of sampling factors, since the limit of ten blocks per MCU no
|
|
longer applies. We could get around this by blithely ignoring the spec's
|
|
limit of ten blocks, but that seems like a bad idea (especially since it makes
|
|
the above problem worse).
|
|
|
|
The upshot is that we need to support both interleaved and noninterleaved JPEG
|
|
formats, since for any given machine and picture size one may be much more
|
|
efficient than the other. However, the non-JPEG format we convert to or from
|
|
will be assumed to be an interleaved format (i.e., it produces or stores all
|
|
the components of a pixel together).
|
|
|
|
I do not think it is necessary for the compressor to be able to output
|
|
partially-interleaved formats (multiple scans, some of which interleave a
|
|
subset of the components). However, the decompressor must be able to read
|
|
such files to conform to the spec.
|
|
|
|
|
|
*** Data formats ***
|
|
|
|
Pipeline steps that work on pixel sample values will use the following data
|
|
structure:
|
|
|
|
typedef something JSAMPLE; a pixel component value, 0..MAXJSAMPLE
|
|
typedef JSAMPLE *JSAMPROW; ptr to a row of samples
|
|
typedef JSAMPROW *JSAMPARRAY; ptr to a list of rows
|
|
typedef JSAMPARRAY *JSAMPIMAGE; ptr to a list of color-component arrays
|
|
|
|
The basic element type JSAMPLE will be one of unsigned char, (signed) char, or
|
|
unsigned short. Unsigned short will be used if samples wider than 8 bits are
|
|
to be supported (this is a compile-time option). Otherwise, unsigned char is
|
|
used if possible. If the compiler only supports signed chars, then it is
|
|
necessary to mask off the value when reading. Thus, all reads of sample
|
|
values should be coded as "GETJSAMPLE(value)", where the macro will be defined
|
|
as "((value)&0xFF)" on signed-char machines and "(value)" elsewhere.
|
|
|
|
With these conventions, JSAMPLE values can be assumed to be >= 0. This should
|
|
simplify correct rounding during subsampling, etc. The JPEG draft's
|
|
specification that sample values run from -128..127 will be accommodated by
|
|
subtracting 128 just as the sample value is copied into the source array for
|
|
the DCT step (this will be an array of signed shorts or longs). Similarly,
|
|
during decompression the output of the IDCT step will be immediately shifted
|
|
back to 0..255. (NB: different values are required when 12-bit samples are in
|
|
use. The code should be written in terms of MAXJSAMPLE and CENTERJSAMPLE,
|
|
which will be #defined as 255 and 128 respectively in an 8-bit implementation,
|
|
and as 4095 and 2048 in a 12-bit implementation.)
|
|
|
|
On compilers that don't support "unsigned short", signed short can be used for
|
|
a 12-bit implementation. To support lossless coding (which allows up to
|
|
16-bit data precision) masking with 0xFFFF in GETJSAMPLE might be necessary.
|
|
(But if "int" is 16 bits then using "unsigned int" is the best solution.)
|
|
|
|
Notice that we use a pointer per row, rather than a two-dimensional JSAMPLE
|
|
array. This choice costs only a small amount of memory and has several
|
|
benefits:
|
|
|
|
* Code using the data structure doesn't need to know the allocated width of
|
|
the rows. This will simplify edge expansion/compression, since we can work
|
|
in an array that's wider than the logical picture width.
|
|
|
|
* The rows forming a component array may be allocated at different times
|
|
without extra copying. This will simplify working a few scanlines at a time,
|
|
especially in smoothing steps that need access to the previous and next rows.
|
|
|
|
* Indexing doesn't require multiplication; this is a performance win on many
|
|
machines.
|
|
|
|
Note that each color component is stored in a separate array; we don't use the
|
|
traditional structure in which the components of a pixel are stored together.
|
|
This simplifies coding of steps that work on each component independently,
|
|
because they don't need to know how many components there are. Furthermore,
|
|
we can read or write each component to a temp file independently, which is
|
|
helpful when dealing with noninterleaved JPEG files.
|
|
|
|
A specific sample value will be accessed by code such as
|
|
GETJSAMPLE(image[colorcomponent][row][col])
|
|
where col is measured from the image left edge, but row is measured from the
|
|
first sample row currently in memory. Either of the first two indexings can
|
|
be precomputed by copying the relevant pointer.
|
|
|
|
|
|
Pipeline steps that work on frequency-coefficient values will use the
|
|
following data structure:
|
|
|
|
typedef short JCOEF; a 16-bit signed integer
|
|
typedef JCOEF JBLOCK[64]; an 8x8 block of coefficients
|
|
typedef JBLOCK *JBLOCKROW; ptr to one horizontal row of 8x8 blocks
|
|
typedef JBLOCKROW *JBLOCKARRAY; ptr to a list of such rows
|
|
typedef JBLOCKARRAY *JBLOCKIMAGE; ptr to a list of color component arrays
|
|
|
|
The underlying type is always a 16-bit signed integer (this is "short" on all
|
|
machines of interest, but let's use the typedef name anyway). These are
|
|
grouped into 8x8 blocks (we should use #defines DCTSIZE and DCTSIZE2 rather
|
|
than "8" and "64"). The contents of a block may be either in "natural" or
|
|
zigzagged order, and may be true values or divided by the quantization
|
|
coefficients, depending on where the block is in the pipeline.
|
|
|
|
Notice that the allocation unit is now a row of 8x8 blocks, corresponding to
|
|
eight rows of samples. Otherwise the structure is much the same as for
|
|
samples, and for the same reasons.
|
|
|
|
On machines where malloc() can't handle a request bigger than 64Kb, this data
|
|
structure limits us to rows of less than 512 JBLOCKs, which would be a picture
|
|
width of 4000 pixels. This seems an acceptable restriction.
|
|
|
|
|
|
On 80x86 machines, the bottom-level pointer types (JSAMPROW and JBLOCKROW)
|
|
must be declared as "far" pointers, but the upper levels can be "near"
|
|
(implying that the pointer lists are allocated in the DS segment).
|
|
To simplify sharing code, we'll have a #define symbol FAR, which expands to
|
|
the "far" keyword when compiling on 80x86 machines and to nothing elsewhere.
|
|
|
|
|
|
The data arrays used as input and output of the DCT transform subroutine will
|
|
be declared using a separate typedef; they could be arrays of "short", "int"
|
|
or "long" independently of the above choices. This would depend on what is
|
|
needed to make the compiler generate correct and efficient multiply/add code
|
|
in the DCT inner loops. No significant speed or memory penalty will be paid
|
|
to have a different representation than is used in the main image storage
|
|
arrays, since some additional value-by-value processing is done at the time of
|
|
creation or extraction of the DCT data anyway (e.g., add/subtract 128).
|
|
|
|
|
|
*** Poor man's object-oriented programming ***
|
|
|
|
It should be pretty clear by now that we have a lot of quasi-independent
|
|
steps, many of which have several possible behaviors. To avoid cluttering the
|
|
code with lots of switch statements, we'll use a simple form of object-style
|
|
programming to separate out the different possibilities.
|
|
|
|
For example, Huffman and arithmetic coding will be implemented as two separate
|
|
modules that present the same external interface; at runtime, the calling code
|
|
will access the proper module indirectly through an "object".
|
|
|
|
We can get the limited features we need while staying within portable C. The
|
|
basic tool is a function pointer. An "object" is just a struct containing one
|
|
or more function pointer fields, each of which corresponds to a method name in
|
|
real object-oriented languages. During initialization we fill in the function
|
|
pointers with references to whichever module we have determined we need to use
|
|
in this run. Then invocation of the module is done by indirecting through a
|
|
function pointer; on most architectures this is no more expensive (and
|
|
possibly cheaper) than a switch, which would be the only other way of making
|
|
the required run-time choice. The really significant benefit, of course, is
|
|
keeping the source code clean and well structured.
|
|
|
|
For example, the interface for entropy decoding (Huffman or arithmetic
|
|
decoding) might look like this:
|
|
|
|
struct function_ptr_struct {
|
|
...
|
|
/* Entropy decoding methods */
|
|
void (*prepare_for_scan) ();
|
|
void (*get_next_mcu) ();
|
|
...
|
|
};
|
|
|
|
typedef struct function_ptr_struct * function_ptrs;
|
|
|
|
The struct pointer is what will actually be passed around. A call site might
|
|
look like this:
|
|
|
|
some_function (function_ptrs fptrs)
|
|
{
|
|
...
|
|
(*fptrs->get_next_mcu) (...);
|
|
...
|
|
}
|
|
|
|
(It might be worth inventing some specialized macros to hide the rather ugly
|
|
syntax for method definition and call.) Note that the caller doesn't know how
|
|
many different get_next_mcu procedures there are, what their real names are,
|
|
nor how to choose which one to call.
|
|
|
|
An important benefit of this scheme is that it is easy to provide multiple
|
|
versions of any method, each tuned to a particular case. While a lot of
|
|
precalculation might be done to select an optimal implementation of a method,
|
|
the cost per invocation is constant. For example, the MCU extraction step
|
|
might have a "generic" method, plus one or more "hardwired" methods for the
|
|
most popular sampling factors; the hardwired methods would be faster because
|
|
they'd use straight-line code instead of for-loops. The cost to determine
|
|
which method to use is paid only once, at startup, and the selection criteria
|
|
are hidden from the callers of the method.
|
|
|
|
This plan differs a little bit from usual object-oriented structures, in that
|
|
only one instance of each object class will exist during execution. The
|
|
reason for having the class structure is that on different runs we may create
|
|
different instances (choose to execute different modules).
|
|
|
|
To minimize the number of object pointers that have to be passed around, it
|
|
will be easiest to have just a few big structs containing all the method
|
|
pointers. We'll actually use two such structs, one for "globally" defined
|
|
methods (applicable to the whole file or to all components of the current
|
|
scan) and one for methods applicable to a single component. There'll be one
|
|
copy of the second kind of struct for each component of the current scan.
|
|
This is necessary so that preselection of an optimal method can be done based
|
|
on component-specific information (like sampling ratios...)
|
|
|
|
Because of this choice, it's best not to think of an "object" as a specific
|
|
data structure. Rather, an "object" is just a group of related methods.
|
|
There would typically be one or more C modules (source files) providing
|
|
concrete implementations of those methods. You can think of the term
|
|
"method" as denoting the common interface presented by some set of functions,
|
|
and "object" as denoting a group of common method interfaces, or the total
|
|
shared interface behavior of a group of modules.
|
|
|
|
|
|
*** Data chunk sizes ***
|
|
|
|
To make the cost of this object-oriented style really minimal, we should make
|
|
sure that each method call does a fair amount of computation. To do that we
|
|
should pass large chunks of data around; for example, the colorspace
|
|
conversion method should process much more than one pixel per call.
|
|
|
|
For many steps, the most natural unit of data seems to be an "MCU row".
|
|
This consists of one complete horizontal strip of the image, as high as an
|
|
MCU. In a noninterleaved scan, an MCU row is always eight samples high (when
|
|
looking at samples) or one 8x8 block high (when looking at coefficients). In
|
|
an interleaved scan, an MCU row consists of all the data for one horizontal
|
|
row of MCUs; this may be from one to four blocks high (eight to thirty-two
|
|
samples) depending on the sampling factors. The height and width of an MCU
|
|
row may be different in each component. (Note that the height and width of an
|
|
MCU row changes at the subsampling and de-subsampling steps. An unsubsampled
|
|
image has the same size in each component. The preceding statements apply to
|
|
the subsampled dimensions.)
|
|
|
|
For example, consider a 1024-pixel-wide image using (2h:2v)(1h:1v)(1h:1v)
|
|
subsampling. In the noninterleaved case, an MCU row of Y would contain 8x1024
|
|
samples or the same number of frequency coefficients, so it would occupy
|
|
8K bytes (samples) or 16K bytes (coefficients). An MCU row of Cb or Cr would
|
|
contain 8x512 samples and occupy half as much space. In the interleaved case,
|
|
an MCU row would contain 16x1024 Y samples, 8x512 Cb and 8x512 Cr samples, so
|
|
a total of 24K (samples) or 48K (coefficients) would be needed. This is a
|
|
reasonable amount of data to expect to retain in memory at one time. (Bear in
|
|
mind that we'll usually need to have several MCU rows resident in memory at
|
|
once, at the inputs and outputs to various pipeline steps.)
|
|
|
|
The worst case is probably (2h:4v)(1h:1v)(1h:1v) interleaving (this uses 10
|
|
blocks per MCU, which is the maximum allowed by the spec). An MCU will then
|
|
contain 32 sample rows worth of Y, so it would occupy 40K or 80K bytes for a
|
|
1024-pixel-wide image. The most memory-intensive step is probably cross-block
|
|
smoothing, for which we'd need 3 MCU rows of coefficients as input and another
|
|
one as output; that would be 320K of working storage. Anything much larger
|
|
would not fit in an 80x86 machine. (To decompress wider pictures on an 80x86,
|
|
we'll have to skip cross-block smoothing or else use temporary files.)
|
|
|
|
This unit is thus a reasonable-sized chunk for passing through the pipeline.
|
|
Of course, its major advantage is that it is a natural chunk size for the MCU
|
|
assembly and disassembly steps to work with.
|
|
|
|
For the entropy (Huffman or arithmetic) encoding/decoding steps, the most
|
|
convenient chunk is a single MCU: one 8x8 block if not interleaved, three to
|
|
ten such blocks if interleaved. The advantage of this is that when handling
|
|
interleaved data, the blocks have the same sequence of component membership on
|
|
each call. (For example, Y,Y,Y,Y,Cb,Cr when using (2h:2v)(1h:1v)(1h:1v)
|
|
subsampling.) The code needs to know component membership so that it can
|
|
apply the right set of compression coefficients to each block. A prebuilt
|
|
array describing this membership can be used during each call. This chunk
|
|
size also makes it easy to handle restart intervals: just count off one MCU
|
|
per call and reinitialize when the count reaches zero (restart intervals are
|
|
specified in numbers of MCU).
|
|
|
|
For similar reasons, one MCU is also the best chunk size for the frequency
|
|
coefficient quantization and dequantization steps.
|
|
|
|
For subsampling and desubsampling, the best chunk size is to have each call
|
|
transform Vk sample rows from or to Vmax sample rows (Vk = this component's
|
|
vertical sampling factor, Vmax = largest vertical sampling factor). There are
|
|
eight such chunks in each MCU row. Using a whole MCU row as the chunk size
|
|
would reduce function call overhead a fraction, but would imply more buffering
|
|
to provide context for cross-pixel smoothing.
|
|
|
|
|
|
*** Compression object structure ***
|
|
|
|
I propose the following set of objects for the compressor. Here an "object"
|
|
is the common interface for one or more modules having comparable functions.
|
|
|
|
Most of these objects can be justified as information-hiding modules.
|
|
I've indicated what information is private to each object/module.
|
|
|
|
Note that in all cases, the caller of a method is expected to have allocated
|
|
any storage needed for it to return its result. (Typically this storage can
|
|
be re-used in successive calls, so malloc'ing and free'ing once per call is
|
|
not reasonable.) Also, much of the context required (compression parameters,
|
|
image size, etc) will be passed around in large common data structures, which
|
|
aren't described here; see the header files. Notice that any object that
|
|
might need to allocate working storage receives an "init" and a "term" call;
|
|
"term" should be careful to free all allocated storage so that the JPEG system
|
|
can be used multiple times during a program run. (For the same reason,
|
|
depending on static initialization of variables is a no-no.)
|
|
|
|
1. Input file conversion to standardized form. This provides these methods:
|
|
input_init: read the file header, report image size & component count.
|
|
get_input_row: read one pixel row, return it in our standard format.
|
|
input_term: finish up at the end.
|
|
In implementations that support multiple input formats, input_init could
|
|
set up an appropriate get_input_row method depending on the format it
|
|
finds. Note that in most applications, the selection and opening of the
|
|
input file will be under the control of the user interface module; and
|
|
indeed the user interface may have already read the input header, so that
|
|
all that input_init may have to do is return previously saved values. The
|
|
behind-the-scenes interaction between this object and the user interface is
|
|
not specified by this architecture.
|
|
(Hides format of input image and mechanism used to read it. This code is
|
|
likely to vary considerably from one implementation to another. Note that
|
|
the color space and number of color components of the source are not hidden;
|
|
but they are used only by the next object.)
|
|
|
|
2. Gamma and color space conversion. This provides three methods:
|
|
colorin_init: initialization.
|
|
get_sample_rows: read, convert, and return a specified number of pixel
|
|
rows (not more than remain in the picture).
|
|
colorin_term: finish up at the end.
|
|
The most efficient approach seems to be for this object to call
|
|
get_input_row directly, rather than being passed the input data; that way,
|
|
any intermediate storage required can be local to this object.
|
|
(get_sample_rows might tell get_input_row to read directly into its own
|
|
output area and then convert in place; or it may do something different.
|
|
For example, conversion in place wouldn't work if it is changing the number
|
|
of color components.) The output of this step is in the standardized
|
|
sample array format shown previously.
|
|
(Hides all knowledge of color space semantics and conversion. Remaining
|
|
modules only need to know the number of JPEG components.)
|
|
|
|
3. Edge expansion: needs only a single method.
|
|
edge_expand: Given an NxM sample array, expand to a desired size (a
|
|
multiple of the MCU dimensions) by duplicating the last
|
|
row or column. Repeat for each component.
|
|
Expansion will occur in place, so the caller must have pre-allocated enough
|
|
storage. (I'm assuming that it is easier and faster to do this expansion
|
|
than it is to worry about boundary conditions in the next two steps.
|
|
Notice that vertical expansion will occur only once, at the bottom of the
|
|
picture, so only horizontal expansion by a few pixels is speed-critical.)
|
|
(This doesn't really hide any information, so maybe it could be a simple
|
|
subroutine instead of a method. Depends on whether we want to be able to
|
|
use alternative, optimized methods.)
|
|
|
|
4. Subsampling: this will be applied to one component at a time.
|
|
subsample_init: initialize (precalculate convolution factors, for
|
|
example). This will be called once per scan.
|
|
subsample: Given a sample array, reduce it to a smaller number of
|
|
samples using specified sampling factors.
|
|
subsample_term: clean up at the end of a scan.
|
|
If the current component has vertical sampling factor Vk and the largest
|
|
sampling factor is Vmax, then the input is always Vmax sample rows (whose
|
|
width is a multiple of Hmax) and the output is always Vk sample rows.
|
|
Vmax additional rows above and below the nominal input rows are also passed
|
|
for use by partial-pixel-averaging sampling methods. (Is this necessary?)
|
|
At the top and bottom of the image, these extra rows are copies of the
|
|
first or last actual input row.
|
|
(This hides whether and how cross-pixel averaging occurs.)
|
|
|
|
5. MCU extraction (creation of a single sequence of 8x8 sample blocks).
|
|
extract_init: initialize as needed. This will be called once per scan.
|
|
extract_MCUs: convert a sample array to a sequence of MCUs.
|
|
extract_term: clean up at the end of a scan.
|
|
Given one or more MCU rows worth of image data, extract sample blocks in the
|
|
appropriate order; pass these off to subsequent steps one MCU at a time.
|
|
The input must be a multiple of the MCU dimensions. It will probably be
|
|
most convenient for the DCT transform, frequency quantization, and zigzag
|
|
reordering of each block to be done as simple subroutines of this step.
|
|
Once a transformed MCU has been completed, it'll be passed off to a
|
|
method call, which will be passed as a parameter to extract_MCUs.
|
|
That routine might either encode and output the MCU immediately, or buffer
|
|
it up for later output if we want to do global optimization of the entropy
|
|
encoding coefficients. Note: when outputting a noninterleaved file this
|
|
object will be called separately for each component. Direct output could
|
|
be done for the first component, but the others would have to be buffered.
|
|
(Again, an object mainly on the grounds that multiple instantiations might
|
|
be useful.)
|
|
|
|
6. DCT transformation of each 8x8 block. This probably doesn't have to be a
|
|
full-fledged method, but just a plain subroutine that will be called by MCU
|
|
extraction. One 8x8 block will be processed per call.
|
|
|
|
7. Quantization scaling and zigzag reordering of the elements in each 8x8
|
|
block. (This can probably be a plain subroutine called once per block by
|
|
MCU extraction; hard to see a need for multiple instantiations here.)
|
|
|
|
8. Entropy encoding (Huffman or arithmetic).
|
|
entropy_encoder_init: prepare for one scan.
|
|
entropy_encode: accepts an MCU's worth of quantized coefficients,
|
|
encodes and outputs them.
|
|
entropy_encoder_term: finish up at end of a scan (dump any buffered
|
|
bytes, for example).
|
|
The data output by this module will be sent to the entropy_output method
|
|
provided by the pipeline controller. (It will probably be worth using
|
|
buffering to pass multiple bytes per call of the output method.) The
|
|
output method could be just write_jpeg_data, but might also be a dummy
|
|
routine that counts output bytes (for use during cut-and-try coefficient
|
|
optimization).
|
|
(This hides which entropy encoding method is in use.)
|
|
|
|
9. JPEG file header construction. This will provide these methods:
|
|
write_file_header: output the initial header.
|
|
write_scan_header: output scan header (called once per component
|
|
if noninterleaved mode).
|
|
write_jpeg_data: the actual data output method for the preceding step.
|
|
write_scan_trailer: finish up after one scan.
|
|
write_file_trailer: finish up at end of file.
|
|
Note that compressed data is passed to the write_jpeg_data method, in case
|
|
a simple fwrite isn't appropriate for some reason.
|
|
(This hides which variant JPEG file format is being written. Also, the
|
|
actual mechanism for writing the file is private to this object and the
|
|
user interface.)
|
|
|
|
10. Pipeline control. This object will provide the "main loop" that invokes
|
|
all the pipeline objects. Note that we will need several different main
|
|
loops depending on the situation (interleaved output or not, global
|
|
optimization of encoding parameters or not, etc). This object will do
|
|
most of the memory allocation, since it will provide the working buffers
|
|
that are the inputs and outputs of the pipeline steps.
|
|
(An object mostly to support multiple instantiations; however, overall
|
|
memory management and sequencing of operations are known only here.)
|
|
|
|
11. Overall control. This module will provide at least two routines:
|
|
jpeg_compress: the main entry point to the compressor.
|
|
per_scan_method_selection: called by pipeline controllers for
|
|
secondary method selection passes.
|
|
jpeg_compress is invoked from the user interface after the UI has selected
|
|
the input and output files and obtained values for all compression
|
|
parameters that aren't dynamically determined. jpeg_compress performs
|
|
basic initialization (e.g., calculating the size of MCUs), does the
|
|
"global" method selection pass, and finally calls the selected pipeline
|
|
control object. (Per-scan method selections will be invoked by the
|
|
pipeline controller.)
|
|
Note that jpeg_compress can't be a method since it is invoked prior to
|
|
method selection.
|
|
|
|
12. User interface; this is the architecture's term for "the rest of the
|
|
application program", i.e., that which invokes the JPEG compressor. In a
|
|
standalone JPEG compression program the UI need be little more than a C
|
|
main() routine and argument parsing code; but we can expect that the JPEG
|
|
compressor may be incorporated into complex graphics applications, wherein
|
|
the UI is much more complex. Much of the UI will need to be written
|
|
afresh for each non-Unix-like platform the compressor is ported to.
|
|
The UI is expected to supply input and output files and values for all
|
|
non-automatically-chosen compression parameters. (Hence defaults are
|
|
determined by the UI; we should probably provide helpful routines to fill
|
|
in recommended defaults.) The UI must also supply error handling
|
|
routines and some mechanism for trace messages.
|
|
(This module hides the user interface provided --- command line,
|
|
interactive, etc. Except for error/message handling, the UI calls the
|
|
portable JPEG code, not the other way around.)
|
|
|
|
13. (Optional) Compression parameter selection control.
|
|
entropy_optimize: given an array of MCUs ready to be fed to entropy
|
|
encoding, find optimal encoding parameters.
|
|
The actual optimization algorithm ought to be separated out as an object,
|
|
even though a special pipeline control method will be needed. (The
|
|
pipeline controller only has to understand that the output of extract_MCUs
|
|
must be built up as a virtual array rather than fed directly to entropy
|
|
encoding and output. This pipeline behavior may also be useful for future
|
|
implementation of hierarchical modes, etc.)
|
|
To minimize the amount of control logic in the optimization module, the
|
|
pipeline control doesn't actually hand over big-array pointers, but rather
|
|
an "iterator": a function which knows how to scan the stored image.
|
|
(This hides the details of the parameter optimization algorithm.)
|
|
|
|
The present design doesn't allow for multiple passes at earlier points
|
|
in the pipeline, but allowing that would only require providing some
|
|
new pipeline control methods; nothing else need change.
|
|
|
|
14. A memory management object. This will provide methods to allocate "small"
|
|
things and "big" things. Small things have to fit in memory and you get
|
|
back direct pointers (this could be handled by direct calls to malloc, but
|
|
it's cleaner not to assume malloc is the right routine). "Big" things
|
|
mean buffered images for multiple passes, noninterleaved output, etc.
|
|
In this case the memory management object will give you room for a few MCU
|
|
rows and you have to ask for access to the next few; dumping and reloading
|
|
in a temporary file will go on behind the scenes. (All big objects are
|
|
image arrays containing either samples or coefficients, and will be
|
|
scanned top-to-bottom some number of times, so we can apply this access
|
|
model easily.) On a platform with virtual memory, the memory manager can
|
|
treat small and big things alike: just malloc up enough virtual memory for
|
|
the whole image, and let the operating system worry about swapping the
|
|
image to disk.
|
|
|
|
Most of the actual calls on the memory manager will be made from pipeline
|
|
control objects; changing any data item from "small" to "big" status would
|
|
require a new pipeline control object, since it will contain the logic to
|
|
ask for a new chunk of a big thing. Thus, one way in which pipeline
|
|
controllers will vary is in which structures they treat as big.
|
|
|
|
The memory manager will need to be told roughly how much space is going to
|
|
be requested overall, so that it can figure out how big a buffer is safe
|
|
to allocate for a "big" object. (If it happens that you are dealing with
|
|
a small image, you'd like to decide to keep it all in memory!) The most
|
|
flexible way of doing this is to divide allocation of "big" objects into
|
|
two steps. First, there will be one or more "request" calls that indicate
|
|
the desired object sizes; then an "instantiate" call causes the memory
|
|
manager to actually construct the objects. The instantiation must occur
|
|
before the contents of any big object can be accessed.
|
|
|
|
For 80x86 CPUs, we would like the code to be compilable under small or
|
|
medium model, meaning that pointers are 16 bits unless explicitly declared
|
|
FAR. Hence space allocated by the "small" allocator must fit into the
|
|
64Kb default data segment, along with stack space and global/static data.
|
|
For normal JPEG operations we seem to need only about 32Kb of such space,
|
|
so we are within the target (and have a reasonable slop for the needs of
|
|
a surrounding application program). However, some color quantization
|
|
algorithms need 64Kb or more of all-in-memory space in order to create
|
|
color histograms. For this purpose, we will also support "medium" size
|
|
things. These are semantically the same as "small" things but are
|
|
referenced through FAR pointers.
|
|
|
|
The following methods will be needed:
|
|
alloc_small: allocate an object of given size; use for any random
|
|
data that's not an image array.
|
|
free_small: release same.
|
|
alloc_medium: like alloc_small, but returns a FAR pointer.
|
|
free_medium: release same.
|
|
alloc_small_sarray: construct an all-in-memory image sample array.
|
|
free_small_sarray: release same.
|
|
alloc_small_barray,
|
|
free_small_barray: ditto for block (coefficient) arrays.
|
|
request_big_sarray: request a virtual image sample array. The size
|
|
of the in-memory buffer will be determined by the
|
|
memory manager, but it will always be a multiple
|
|
of the passed-in MCU height.
|
|
request_big_barray: ditto for block (coefficient) arrays.
|
|
alloc_big_arrays: instantiate all the big arrays previously requested.
|
|
This call will also pass some info about future
|
|
memory demands, so that the memory manager can
|
|
figure out how much space to leave unallocated.
|
|
access_big_sarray: obtain access to a specified portion of a virtual
|
|
image sample array.
|
|
access_big_barray: ditto for block (coefficient) arrays.
|
|
free_big_sarray: release a virtual sample array.
|
|
free_big_barray: ditto for block (coefficient) arrays.
|
|
|
|
alloc_big_arrays will be called by the pipeline controller, which does
|
|
most of the memory allocation anyway. The only reason for having separate
|
|
request calls is to allow some of the other modules to get big arrays.
|
|
The pipeline controller is required to give an upper bound on total future
|
|
small-array requests, so that this space can be discounted. (A fairly
|
|
conservative estimate will be adequate.) Future small-object requests
|
|
aren't counted; the memory manager has to use a slop factor for those.
|
|
10K or so seems to be sufficient. (In an 80x86, small objects aren't an
|
|
issue anyway, since they don't compete for far-heap space. "Medium"-size
|
|
objects will have to be counted separately.)
|
|
|
|
The distinction between sample and coefficient array routines is annoying,
|
|
but it has to be maintained for machines in which "char *" is represented
|
|
differently from "int *"... on byte-addressable machines some of these
|
|
methods could point to the same code.
|
|
|
|
The array routines will operate on only 2-D arrays (one component at a
|
|
time), since different components may require different-size arrays.
|
|
|
|
(This object hides the knowledge of whether virtual memory is available,
|
|
as well as the actual interface to OS and library support routines.)
|
|
|
|
Note that any given implementation will presumably contain only one
|
|
instantiation of input file header reading, overall control, user interface,
|
|
and memory management. Thus these could be called as simple subroutines,
|
|
without bothering with an object indirection. This is essential for overall
|
|
control (which has to initialize the object structure); I'm undecided whether
|
|
to impose objectness on the other three.
|
|
|
|
|
|
*** Decompression object structure ***
|
|
|
|
I propose the following set of objects for decompression. The general
|
|
comments at the top of the compression object section also apply here.
|
|
|
|
1. JPEG file scanning. This will provide these methods:
|
|
read_file_header: read the file header, determine which variant
|
|
JPEG format is in use, read everything through SOF.
|
|
read_scan_header: read scan header (up through SOS). This is called
|
|
after read_file_header and again after each scan;
|
|
it returns TRUE if it finds SOS, FALSE if EOI.
|
|
read_jpeg_data: fetch data for entropy decoder.
|
|
read_scan_trailer: finish up after one scan, prepare for another call
|
|
of read_scan_header (may be a no-op).
|
|
read_file_trailer: finish up at end of file (probably a no-op).
|
|
The entropy decoder must deal with restart markers, but all other JPEG
|
|
marker types will be handled in this object; useful data from the markers
|
|
will be extracted into data structures available to subsequent routines.
|
|
Note that on exit from read_file_header, only the SOF-marker data should be
|
|
assumed valid (image size, component IDs, sampling factors); other data
|
|
such as Huffman tables may not appear until after the SOF. The overall
|
|
image size and colorspace can be determined after read_file_header, but not
|
|
whether or how the data is interleaved. (This hides which variant JPEG
|
|
file format is being read. In particular, for JPEG-in-TIFF the read_header
|
|
routines might not be scanning standard JPEG markers at all; they could
|
|
extract the data from TIFF tags. The user interface will already have
|
|
opened the input file and possibly read part of the header before
|
|
read_file_header is called.)
|
|
|
|
NOTE: for JFIF/raw-JPEG file format, the read_jpeg_data routine is actually
|
|
supplied by the user interface; the jrdjfif module uses read_jpeg_data
|
|
internally to scan the input stream. This makes it possible for the user
|
|
interface module to single-handedly implement special applications like
|
|
reading from a non-stdio source. For JPEG-in-TIFF format, the need for
|
|
random access will make it impossible for this to work; hence the TIFF
|
|
header module will probably override the UI read_jpeg_data routine.
|
|
Non-stdio input from a TIFF file will require extensive surgery to the TIFF
|
|
header module, if indeed it is practical at all.
|
|
|
|
2. Entropy (Huffman or arithmetic) decoding of the coefficient sequence.
|
|
entropy_decoder_init: prepare for one scan.
|
|
entropy_decode: decodes and returns an MCU's worth of quantized
|
|
coefficients per call.
|
|
entropy_decoder_term: finish up after a scan (may be a no-op).
|
|
This will read raw data by calling the read_jpeg_data method (I don't see
|
|
any reason to provide a further level of indirection).
|
|
(This hides which entropy encoding method is in use.)
|
|
|
|
3. Quantization descaling and zigzag reordering of the elements in each 8x8
|
|
block. (This can probably be a plain subroutine called once per block;
|
|
hard to see a need for multiple instantiations here.)
|
|
|
|
4. MCU disassembly (conversion of a possibly interleaved sequence of 8x8
|
|
blocks back to separate components in pixel map order).
|
|
disassemble_init: initialize. This will be called once per scan.
|
|
disassemble_MCU: Given an MCU's worth of dequantized blocks,
|
|
distribute them into the proper locations in a
|
|
coefficient image array.
|
|
disassemble_term: clean up at the end of a scan.
|
|
Probably this should be called once per MCU row and should call the
|
|
preceding two objects repeatedly to obtain the row's data. The output is
|
|
always a multiple of an MCU's dimensions.
|
|
(An object on the grounds that multiple instantiations might be useful.)
|
|
|
|
5. Cross-block smoothing per JPEG section 13.10 or a similar algorithm.
|
|
smooth_coefficients: Given three block rows' worth of a single
|
|
component, emit a smoothed equivalent of the
|
|
middle row. The "above" and "below" pointers
|
|
may be NULL if at top/bottom of image.
|
|
The pipeline controller will do the necessary buffering to provide the
|
|
above/below context. Smoothing will be optional since a good deal of
|
|
extra memory is needed to buffer the additional block rows.
|
|
(This object hides the details of the smoothing algorithm.)
|
|
|
|
6. Inverse DCT transformation of each 8x8 block. (This can be a plain
|
|
subroutine processing one block per call.)
|
|
|
|
7. De-subsampling and smoothing: this will be applied to one component at a
|
|
time. Note that cross-pixel smoothing, which was a separate step in the
|
|
prototype code, will now be performed simultaneously with expansion.
|
|
unsubsample_init: initialize (precalculate convolution factors, for
|
|
example). This will be called once per scan.
|
|
unsubsample: Given a sample array, enlarge it by specified sampling
|
|
factors.
|
|
unsubsample_term: clean up at the end of a scan.
|
|
If the current component has vertical sampling factor Vk and the largest
|
|
sampling factor is Vmax, then the input is always Vk sample rows (whose
|
|
width is a multiple of Hk) and the output is always Vmax sample rows.
|
|
Vk additional rows above and below the nominal input rows are also passed
|
|
for use in cross-pixel smoothing. At the top and bottom of the image,
|
|
these extra rows are copies of the first or last actual input row.
|
|
(This hides whether and how cross-pixel smoothing occurs.)
|
|
|
|
8. Cropping to the original pixel dimensions (throwing away duplicated
|
|
pixels at the edges). This won't be a separate object, just an
|
|
adjustment of the nominal image size in the pipeline controller.
|
|
|
|
9. Color space reconversion and gamma adjustment.
|
|
colorout_init: initialization. This will be passed the component
|
|
data from read_file_header, and will determine the
|
|
number of output components.
|
|
color_convert: convert a specified number of pixel rows. Input and
|
|
output are image arrays of same size but possibly
|
|
different numbers of components.
|
|
colorout_term: cleanup (probably a no-op except for memory dealloc).
|
|
In practice will always be given an MCU row's worth of pixel rows, except
|
|
at the bottom where a smaller number of rows may be left over. Note that
|
|
this object works on all the components at once.
|
|
(Hides all knowledge of color space semantics and conversion. Remaining
|
|
modules only need to know the number of JPEG and output components.)
|
|
|
|
10. Color quantization (used only if a colormapped output format is requested).
|
|
We use two different strategies depending on whether one-pass (on-the-fly)
|
|
or two-pass quantization is requested. Note that the two-pass interface
|
|
is actually designed to let the quantizer make any number of passes.
|
|
color_quant_init: initialization, allocate working memory. In 1-pass
|
|
quantization, should call put_color_map.
|
|
color_quantize: convert a specified number of pixel rows. Input
|
|
and output are image arrays of same size, but input
|
|
is N coefficients and output is only one. (Used only
|
|
in 1-pass quantization.)
|
|
color_quant_prescan: prescan a specified number of pixel rows in
|
|
2-pass quantization.
|
|
color_quant_doit: perform multi-pass color quantization. Input is a
|
|
"big" sample image, output is via put_color_map and
|
|
put_pixel_rows. (Used only in 2-pass quantization.)
|
|
color_quant_term: cleanup (probably a no-op except for memory dealloc).
|
|
For one-pass quantization the image is simply processed by color_quantize,
|
|
a few rows at a time. For two-pass quantization, the pipeline controller
|
|
accumulates the output of color_convert into a "big" sample image. The
|
|
color_quant_prescan method is invoked during this process so that the
|
|
quantizer can accumulate statistics. At the end of the image,
|
|
color_quant_doit is called; it must rescan the "big" image and pass
|
|
converted data to the output module. Additional scans of the image could
|
|
be made before the output pass is done (in fact, prescan could be a no-op).
|
|
As with entropy parameter optimization, the pipeline controller actually
|
|
passes an iterator function rather than direct access to the big image.
|
|
NOTE: it might be better to do this on the internal color space instead of
|
|
RGB? If so, it would need to be performed one step earlier.
|
|
(Hides color quantization algorithm.)
|
|
|
|
11. Writing of the desired image format.
|
|
output_init: produce the file header given data from read_file_header.
|
|
put_color_map: output colormap, if any (called by color quantizer).
|
|
put_pixel_rows: output image data in desired format.
|
|
output_term: finish up at the end.
|
|
Note that whether colormapping is needed will be determined by the user
|
|
interface object prior to method selection. In implementations that
|
|
support multiple output formats, the actual output format will also be
|
|
determined by the user interface.
|
|
(Hides format of output image and mechanism used to write it. Note that
|
|
several other objects know the color model used by the output format. The
|
|
actual mechanism for writing the file is private to this object and the
|
|
user interface.)
|
|
|
|
12. Pipeline control. This object will provide the "main loop" that invokes
|
|
all the pipeline objects. Note that we will need several different main
|
|
loops depending on the situation (interleaved input or not, whether to
|
|
apply cross-block smoothing or not, etc). We may want to divvy up the
|
|
pipeline controllers into two levels, one that retains control over the
|
|
whole file and one that is invoked per scan.
|
|
This object will do most of the memory allocation, since it will provide
|
|
the working buffers that are the inputs and outputs of the pipeline steps.
|
|
(An object mostly to support multiple instantiations; however, overall
|
|
memory management and sequencing of operations are known only here.)
|
|
|
|
13. Overall control. This module will provide at least two routines:
|
|
jpeg_decompress: the main entry point to the decompressor.
|
|
per_scan_method_selection: called by pipeline controllers for
|
|
secondary method selection passes.
|
|
jpeg_decompress is invoked from the user interface after the UI has
|
|
selected the input and output files and obtained values for all
|
|
user-specified options (e.g., output file format, whether to do block
|
|
smoothing). jpeg_decompress calls read_file_header, performs basic
|
|
initialization (e.g., calculating the size of MCUs), does the "global"
|
|
method selection pass, and finally calls the selected pipeline control
|
|
object. (Per-scan method selections will be invoked by the pipeline
|
|
controller.)
|
|
Note that jpeg_decompress can't be a method since it is invoked prior to
|
|
method selection.
|
|
|
|
14. User interface; this is the architecture's term for "the rest of the
|
|
application program", i.e., that which invokes the JPEG decompressor.
|
|
The UI is expected to supply input and output files and values for all
|
|
operational parameters. The UI must also supply error handling routines.
|
|
At the moment I can't think of any nonfatal errors the JPEG code is likely
|
|
to report, so a single report-this-error-and-exit method should be
|
|
sufficient.
|
|
(This module hides the user interface provided --- command line,
|
|
interactive, etc. Except for error handling, the UI calls the portable
|
|
JPEG code, not the other way around.)
|
|
|
|
15. A memory management object. This will be identical to the memory
|
|
management for compression (and will be the same code, in combined
|
|
programs). See above for details.
|
|
|
|
|
|
*** Initial method selection ***
|
|
|
|
The main ugliness in this design is the portion of startup that will select
|
|
which of several instantiations should be used for each of the objects. (For
|
|
example, Huffman or arithmetic for entropy encoding; one of several pipeline
|
|
controllers depending on interleaving, the size of the image, etc.) It's not
|
|
really desirable to have a single chunk of code that knows the names of all
|
|
the possible instantiations and the conditions under which to select each one.
|
|
|
|
The best approach seems to be to provide a selector function for each object
|
|
(group of related method calls). This function knows about each possible
|
|
instantiation of its object and how to choose the right one; but it doesn't
|
|
know about any other objects.
|
|
|
|
Note that there will be several rounds of method selection: at initial startup,
|
|
after overall compression parameters are determined (after the file header is
|
|
read, if decompressing), and one in preparation for each scan (this occurs
|
|
more than once if the file is noninterleaved). Each object method will need
|
|
to be clearly identified as to which round sets it up.
|
|
|
|
|
|
*** Implications of DNL marker ***
|
|
|
|
Some JPEG files may use a DNL marker to postpone definition of the image
|
|
height (this would be useful for a fax-like scanner's output, for instance).
|
|
In these files the SOF marker claims the image height is 0, and you only
|
|
find out the true image height at the end of the first scan.
|
|
|
|
We could handle these files as follows:
|
|
1. Upon seeing zero image height, replace it by 65535 (the maximum allowed).
|
|
2. When the DNL is found, update the image height in the global image
|
|
descriptor.
|
|
This implies that pipeline control objects must avoid making copies of the
|
|
image height, and must re-test for termination after each MCU row. This is
|
|
no big deal.
|
|
|
|
In situations where image-size data structures are allocated, this approach
|
|
will result in very inefficient use of virtual memory or
|
|
much-larger-than-necessary temporary files. This seems acceptable for
|
|
something that probably won't be a mainstream usage. People might have to
|
|
forgo use of memory-hogging options (such as two-pass color quantization or
|
|
noninterleaved JPEG files) if they want efficient conversion of such files.
|
|
(One could improve efficiency by demanding a user-supplied upper bound for the
|
|
height, less than 65536; in most cases it could be much less.)
|
|
|
|
Alternately, we could insist that DNL-using files be preprocessed by a
|
|
separate program that reads ahead to the DNL, then goes back and fixes the SOF
|
|
marker. This is a much simpler solution and is probably far more efficient.
|
|
Even if one wants piped input, buffering the first scan of the JPEG file
|
|
needs a lot smaller temp file than is implied by the maximum-height method.
|
|
For this approach we'd simply treat DNL as a no-op in the decompressor (at
|
|
most, check that it matches the SOF image height).
|
|
|
|
We will not worry about making the compressor capable of outputting DNL. Note
|
|
that something similar to the first scheme above could be applied if anyone
|
|
ever wants to make that work.
|
|
|
|
|
|
*** Notes for MS-DOS implementors ***
|
|
|
|
The standalone cjpeg and djpeg applications can be compiled in "small" memory
|
|
model, at least at the moment; as the code grows we may be forced to switch to
|
|
"medium" model. (Small = both code and data pointers are near by default;
|
|
medium = far code pointers, near data pointers.) Medium model will slow down
|
|
calls through method pointers, but I don't think this will amount to any
|
|
significant speed penalty.
|
|
|
|
When integrating the JPEG code into a larger application, it's a good idea to
|
|
stay with a small-data-space model if possible. An 8K stack is much more than
|
|
sufficient for the JPEG code, and its static data requirements are less than
|
|
1K. When executed, it will typically malloc about 10K worth of near heap
|
|
space (and lots of far heap, but that doesn't count in this calculation).
|
|
This figure will vary depending on image size and other factors, but figuring
|
|
20K should be more than sufficient. Thus you have about 35K available for
|
|
other modules' static data and near heap requirements before you need to go to
|
|
a larger memory model. The C library's static data will account for several K
|
|
of this, but that still leaves a good deal for your needs. (If you are tight
|
|
on space, you could reduce JPEG_BUF_SIZE from 4K to 1K to save 3K of near heap
|
|
space.)
|
|
|
|
As the code is improved, we will endeavor to hold the near data requirements
|
|
to the range given above. This does imply that certain data structures will
|
|
be allocated as FAR although they would fit in near space if we assumed the
|
|
JPEG code is stand-alone. (The LZW tables in jrdgif/jwrgif are examples.)
|
|
To make an optimal implementation, you might want to move these structures
|
|
back to near heap if you know there is sufficient space.
|
|
|
|
|
|
*** Potential optimizations ***
|
|
|
|
For colormapped input formats it might be worthwhile to merge the input file
|
|
reading and the colorspace conversion steps; in other words, do the colorspace
|
|
conversion by hacking up the colormap before inputting the image body, rather
|
|
than doing the conversion on each pixel independently. Not clear if this is
|
|
worth the uglification involved. In the above design for the compressor, only
|
|
the colorspace conversion step ever sees the output of get_input_row, so this
|
|
sort of thing could be done via private agreement between those two modules.
|
|
|
|
Level shift from 0..255 to -128..127 may be done either during colorspace
|
|
conversion, or at the moment of converting an 8x8 sample block into the format
|
|
used by the DCT step (which will be signed short or long int). This could be
|
|
selectable by a compile-time flag, so that the intermediate steps can work on
|
|
either signed or unsigned chars as samples, whichever is most easily handled
|
|
by the platform. However, making sure that rounding is done right will be a
|
|
lot easier if we can assume positive values. At the moment I think that
|
|
benefit is worth the overhead of "& 0xFF" when reading out sample values on
|
|
signed-char-only machines.
|