Title: Optimal policies for queuing systems with periodic review
Full Citation
Permanent Link: http://ufdc.ufl.edu/UF00097765/00001
 Material Information
Title: Optimal policies for queuing systems with periodic review
Physical Description: Book
Language: English
Creator: Magazine, Michael Jay, 1943-
Copyright Date: 1969
Genre: bibliography   ( marcgt )
non-fiction   ( marcgt )
 Record Information
Bibliographic ID: UF00097765
Volume ID: VID00001
Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
Resource Identifier: alephbibnum - 001036266
oclc - 18278439
notis - AFB8640


This item has the following downloads:

PDF ( 2 MBs ) ( PDF )

Full Text







Roger Eric



I would like to express my sincere gratitude to the

members of my supervisory committee and in particular its

Chairman, Professor Michael E. Thomas. His concern and

inspirations were invaluable toward the completion of this

dissertation. Also, I must thank Professor George Nemhauser,

first for original assistance in formulatin: the problem and

secondly, for his continual consulting.

One individual who has been foremost in never refusing

to offer any type of assistance is my office mate, John McKinney.

In spite of having to overcome the same problems as myself,

he always seemed to be there when needed.

I also would like to thank Lydia Gonzalez for so ably

typing my draft, and Mrs. Terrie Duff for her typing of my

final copy, overcoming difficulties of time and legibility.

Last, but most, I would like to thank my wife, Joan, who,

besides her forcitude and concern, has made im dedication

page possible.



ACKNOWLEDGIIENTS . . . . . . . . ... iii

ABSTRACT . . . . . . . . . . . .



II. SINGLE SERVER MODEL. . . . . . . 5


IV. MULTI-SERVER MODEL . . . . . .. 44



I. PIECEWISE CONVEXITY . . . . . .. 56


BIBLIOGPAPHY . . . . . . . . . . . 67

BICGRAPHICCAL SKETCH . . . . . . . .. 69

Abstract of Dissertation Presented to the Graduate Council
in Partial Fulfillment of the Requirements for the
Degree of Doctor of Philosophy



Michael J. Magazine

August, 1969

Chairman: Michael E. Thomas
Major Department: Industrial and Systems Engineering

The general problem cf finding optimal policies for oper-

ating queuing systems is investigated in order to determine

when to open or close servers so as to minimize cost. The

queue size is observed at equally spaced intervals cf time

and the structure of the optimal policy is determined.

The main restriction made, at first, is to limit the model

to a single server system. Justification for a form of an opti-

mal policy is given. Then the proofs that this policy is in

fact optimal for the finite horizon, infinite horizon, and

averaging cost criterion are given. Also, conditions for

determining that the server should never be closed with custo-

mers in queue are presented.

Two computational schemes for determining the optimal

policy are presented. The first, utilizing system parameters,

minimizes average cost by computing the expected waiting time.

The other solves the infinite horizon discounted cost criterion

by direct use of Howard's :!arkov programming algorithm.

The problem is then generalized to allow for a fixed num-

ber of servers. Most of the results from the single server

follow immediately. The main difficulty is, therefore, to

establish and justify the desired rule so that it is analo-

gous with the single server and is still as general as possible.

The primary mathematical concept used in solving the prob-

lem is dynamic programming. This technique yields the solution

to the finite and infinite discounted cost criteria in both

models presented. Some of the results from replacement theory

help in solving the averaging case.

Finally, the conclusion shows how this work is an extension

of the related work in the field, and points out several exten-

sions which naturally follow. The fact that the form of the

optimal policy is both intuitively appealing and easy to imple-

ment makes the results useful in many queuing and related

systems, such as inventory and replacement models.




An important problem to consider is the economic behavior

of a stochastic service system. Most research in queuing

analysis has dealt with the underlying probabilistic structure

rather than identifying optimal policies for the operation

of the system. Within the operation of a service station

there is a fraction cf time in which the server is idle. This

idle fraction can be eliminated in one of two ways. Firs:,

the server can carry out auxiliary duties during this idle

fraction or, as an alternative, there can be a partial or

complete elimination of the station. We are concerned which

the latter, i.e., when to open or close our station in --der

that some objective is optimized. The solution to this prob-

lem is by no means immediately obvious,especially when there

is a statistical input of customers. In this paper we will

discuss a reasonable form for an optimal solution, prove

under what conditions this solution exists and discuss exten-

sions including computational refinements for this type of


Previous Pesults

The justification for this work stems from the similarity

between queuing and inventory models. Much work has been

done using explicit cost functions on inventory models17

in particular getting optimal policies of a particular struc-

ture, such as the s-S policy. Because of this similarity in

structure it seemed reasonable to attempt to fit explicit

cost functions to the model with the hope of finding opti-

mal policies similar in structure to the s-S policy.

The mathematical background for this dissertation comes

first from dynamic programming2'13 and then from many papers
on Markovian Decision Processes.9,0 Many of these works are
discussed in a survey paper by the author. In a paper on

replacement policies which is an application of sequential

decision process, Derman5 defines a class of rules as con-

trol limit rules. The form of this rule was helpful in ob-

taining optimal rules for the queuing mcdel.

The earliest paper on optimization of queuing systems
was by Edie. He uses optimization techniques in determin-

ing the best level of service based on traffic volume at toll

booths. Then the number of toll booths required at any time

of day can be established. Edie's results provided a minimum

number of toll collectors providing acceptable service with

relief for the toll collectors.

A paper by Yadin and Naorl7 was an important inspira-

tion for this work. They dealt with the partial elimination

of a single service station under a (O,R) doctrine. This is

where the server when closed remains closed until R customers

are present in the system and remains open until the system

is empty. They give properties of this system under this

rule and also give computational aspects leading to a closed

form solution for the value of R.

Most recently Heyman and Sobell5 have recognized the

importance of explicit cost functions for stochastic service

systems. Heyman considers a single channel queue with Poisson

arrivals and arbitrary service time distribution. Upon com-

pletion of arrival and service epochs he shows that for infi-

nite horizon models there exists an optimal stationary policy

of particular form. He also finds an explicit cost function

and thereby is able to find an optimal policy as a function

of the parameters of the system. Heyman next considers a

two-channel model, one of which is always open, with the

added assumption that the service time distribution is expo-

nential. He again shows what form the optimal policy must

take and uses Jewell's Markov Renewal Programming technique

to find the policy.

Sobel generalizes the problem by making the arrival dis-

tribution arbitrary. Also, where Heyman uses a linear hold-

ing cost, Sobel uses a very general holding cost. He then

shows existence for the same type of optimal policy as does


General Problem Statement

Consider "customers" arriving at a service facility from

an infinite population according to some statistically steady

input stream with distribution function A(t). Also, assume

there are at most s servers, each with distribution of ser-

vice time Bk(t), k = 1, 2, ..., s. The system is observed

at discrete times (t, t2 ...). We are interested in first

showing the existence and then finding a policy of form (A)

so as to minimize some cost structure, where form (A) is:

(A) for each k (k = 1, 2, ... s) identify a unique

ordered pair [a*(t), b*(t)] such that if i, the number

in the system, falls between a*(t) and b*(t) a specific

action will be determined. If i does not fall between

these values then no action will be taken. By an action

we mean how many servers should be opened or closed .

The form of the optimal policy for each specific prob-

lem will be of this structure and will also inherently

make it clear how many servers should be opened or closed.

This differs from the aforementioned works because of the

following refinements:

(1) Assume exponential inter-arrival and service distri-


(2) Allow for a fixed and finite number of servers.

(3) Show the existence of an optimal policy for finite

horizon, infinite horizon and averaging cost criteria.

(4) Make the decision points at the end of time periods

of equal length. This is a major difference between

this work and the others. Observing at the end of

equal time intervals causes many difficulties that

are not encountered when observation points are the

completion of arrival or service epochs. These will

be pointed our later in the text.



Let us first consider a modification of this problem

which is far more tractable.

A Specific Problem

As a specific problem let us make the following changes

from the general problem formulation.

(I) Assume there is just one server. This is done

solely to simplify the problem,and hopefully will give insight

to the rrultiserver case.

(II) Assume the system has no bounds on capacity. But

also assume there is only a finite number of states for which

decisions will be made, i.e. when observed with more than M

customers, for some finite M, the decision will always be tc

open if we are closed. This is not a major restriction and

the reason for it will be pointed out later in the text.

(III) Assume stationary solutions. Blackwell assures

that both a* and b* will not be functions of time, in an

infinite horizon model, because of the finiteness of the

action space (here, only two possible decisions).

(IV) Assume both the times between arrivals and services

are distributed exponentially. Some comments are needed to

justify this assumption. A desirable characteristic of a

real-life service system is that the monitoring or checking

times are not too frequent. In fact, we would like to ob-

serve our system at equal time intervals (once a minute, hour,

day, etc.). To ensure the problem is solvable the sequence

of states must form a Markov Chain (number of customers at

our service station must be independent of the past history

of the process); that is, there must be stationary transition

probabilities. This implies that the number of arrivals and

services must only depend on the length of the time period.

In order not to increase the dimensionality cf the state space,

A(t) and B(t) are restricted to be exponential distribution

functions. If, for example, B(t) has a general probability

distribution, to ensure the Mlarkovian property it would be

necessary to observe the system at the completion of each

service when it was open. We have chosen to be more general

in our observation points than in the arrival and service

distributions. However, much of the analysis and many of the

results can be used for the other, more general, case.

The specific problem can now be stated. Customers arrive

before a single server according to the Poisson probability

law. The time between services is exponential. The system

is observed at equal time intervals t = 1, 2, ... We are

interested in first showing the existence and then finding

a policy of form (A') so as to minimize some cost function;

where (A') is:

(A') identify a unique pair (j*, i*) with the interpre-

tation that if and only if the server is open when

observed and there are at most j* customers present will

the system be closed and if and only if the server is

closed and there are at least i* customers present when

observed will the server be opened.

Note that this policy could be expressed precisely in the form

of (A) from Chapter I. If for k = 0, no servers open, we

choose (a*, b*) to be(i*, -) then the action will be to open

the server if the number in the system is between a* and b*,

and no action would be taken otherwise. Similarly, if k = 1

and (a*, b*) is chosen as (--, j*) then the server will only

be closed when the number of customers is between a* and b*.

This differs with Yadin and Naor in that they assumed

a (O,R) doctrine and computationally found the optimal value

of R.


The properties of this model can now be given as follows:

(a) The Arrival Pattern -- Customers arrive so that the

time between successive arrivals is distributed exponentially.

(b) The Service Mechanism -- Customers are served in

order of their arrival so that the time between successive

services is distributed exponentially.

(c) Cost Structure --

(i) A Constant cost of opening a closed server

(ii) B Constant cost of closing an open server

(iii) C Cost of operating the server for one time


(iv) K(i) Holding cost, i.e., cost incurred when

i customers are observed in the system.

These costs may or may not be discounted with discount

factor a.

The decision process is as follows. The system is observed

at times t = 1, 2, ... and the state of the system Xt is noted.

Xt = i implies there are i customers in the system at time t.

The decision is then made to either leave the server in its

present state (opened or closed) or to open a closed server or

to close an open server, or if X > M and the server is closed it

must be opened.

The sequence of states {X.} forms a Markov Chain and changes

of states occur according to the following probabilistic struc-


pij = Pr {X = jXt = i and the server is closed)

qi = Pr {Xt+1 = jIXt = i and the server is openI

Existence of a Control Limit Rule

Consider a single channel system with the aforementioned

characteristics. We would like to establish the existence of

a control limit rule. This is, if the server is closed show

that there exists a unique number i*, such that if the number

of customers observed in the system is no less than i* then

open the server. Further, if the number observed is less

than i* then keep the server closed. This rule has intuitive

appeal, saying, that if the server is not open with, say, 7

customers in queue then certainly it will not be open with,

say, 5.

It is desirable to show the existence of this number for

an averaging objective function. In doing so we obtain simi-

lar results for both the discounted finite and infinite hori-

zon models.

Dynamic Programming Formulation

Consider the solution to the finite horizon model using

dynamic programming. Define,

k (i,a) = minimum expected cost with n time periods
remaining when i customers are observed in

the queue using discount factor c-, and k = 0,

1 depending on whether the server is closed

or open respectively.

Then the recursive equations are
o 0
o (i,a) = min K(i) + aZ p. 9 (j,a)
n j= ij n -1

K(i) + A + C + a q.j (j a)

where the alternatives are to keep the server closed or to open


And 4I (i,a) =min K K(i) + B + a Z -(a)
d n j=0

K(i) + C + C T a.. 1 )
j=0 -

with alternatives closing the server or keeping it open, where

it is assumed that
0 1
0 (m,a) = K(C) + A + C + Z qmj m-j,a) for m > M
j=0 mj


0(i,.) = K(i) k = 0, 1

i.e., the terminal cost of our process is the holding cost

for the number of customers observed. In addition, assume

(i) that K(i) is a nondecreasing function of i, a

reasonable assumption and one often made in inven-

tory or replacement analysis.

and (ii) that for any nondecreasing function h(j), p..h(j)
and I qi.h(j) are nondecreasing functions of i.
This says, given some subset of states K = {k,

k+l, ...} the Chance of entering K is nondecreasing

in i.

Consider the server closed. For n = 1 we have

S(i, a) = min K(i) +a Z pi K(j)

K(i) + A + C +ca q K(j)
j=0 1j
From assumptions (i) and (ii) we know both of these alternatives
and also, (i,a) are nondecreasing. If at i = 0, the second

alternative is chosen then we must ensure it must be chosen

for all i (always open server). If at i = 0 the first alterna-

tive is chosen then these t,'o nondecreasing curves may cross in

at most one place, the i* being sought. If they don't cross

at all before M. then the server will never be opened until

at least M customers are present. The possibilities are

illustrated in Figure 1. To ensure case c cannot occur an

additional assumption must be made.


a i


a Never open server

c Undesirable case
- - cost for closed server

cost for open server




d i* i

b Never close server

d Unique opening point

Figure 1: Associated costs for two alternatives




We should comment here how Heyman's model, that of check-

ing after each arrival,fails to encounter this difficulty.

Let us say that case c denotes the graphs of the curves in

Equation 1. Let i be the first crossing point and i2(>i1)

the second, that is keep the server closed for i
i >i2 and open otherwise. If the system is observed at i
and then at each arrival thereafter, as Heyman does, as soon

as il is reached the server will be opened and the state i > i2

for a closed server will never be reached. Checking at dis-

crete time points, however, may result for some t in

Xt = i for i < i1

and Xt+1 = j for j > i2

resulting in keeping the server closed at both points, and

hence his rule of always opening after iI may not hold.

Additional Restriction

As so often happens in this type of problem when a barrier

is reached, an additional condition must be imposed. By anal-

ogy to the definition of convexity we impose the following


8K(j) + (1-8) K(i) > K.(j + (l-e)i) for

8e[0,l] and the argument for K(.) integral (2)

For obvious reasons we call (2) the convex property. It would

be possible to achieve the same results by defining the envelope

of the function as convex. However, there is very little addi-

tional computation in analyzing the discrete function.

Uniqueness Proof for n = 1

Recalling that

S(i,a) = min K(i) + a Z .K(j)
'1 j=0o

K(i) + A + C + e F q..K(j)

it must be determined when

K(i) + a Z p. K(j) (-) K(i) + A + C + a q jK(j)
j=0 j =0


SC + A
Z K(j) [pij q ij] A
j=0 _

This will be done by showing that

Z K(j) [ij qij]

is a nondecreasing function of i and hence can cross the con-

stant line A at most once.

Let x and y be the number cf arrivals and services,

respectively, in a given time period.


Z p K(j) = Z Pr (number of arrivals equals x) "K(i--x)
j=0 x

= E K(i + x) = E E K(i + x)
x x y

and also,

Z qi. K(j) = I: Pr(x arrivals) Pr (y services)
j=0 x y

= E E K(i+x-y)
x y


K(n) = 0 for n < 0.

Since K is nondecreasing

D. E E [K(i+x) K(i+x-y)] > 0[1]

Z K(j) [Pij

- q i >_ 0
ij -

Now, allowing j = i + h, (2) can be rewritten as

6K(i + h) + (1 e) K(i) > K(ei + eh + i ei)

= K(i + oh)

6K(i + h) gK(i) > K(i + ah) K(i)

To say this we must ensure that both 7 Pi K(j) and
7 q .K(j) are finite. To show convergence employ the ratio
test as follows:

SPijK(j) =

S (J-i) e-\
ji (j K(j)

so that

Pj-i+1 -;\
,i+ (j-i+1)!
j-i+1 -.\
Pi, K(j) = (-i+l
1,i(j-i) a

k(j+l) and

nd K(j) and the limit as j-- of

piz11K(j+l) N K(j+l) must be <1. This holds for
P K(j) (T-i+1) K(j)
a wide class of functions such as exponential and polynomials
and will be assumed true. Certainly, then, qij.K(j) will also
converge as it is less than or equal to 7 pijK(j).

or as an alternate definition

K(i + h) K(i) > [K(i + h) K(i)] (2')

which must hold for all 0 < <_ 1 and eh integer. In particular

it must hold for Oh = 1, i.e.,

K(i + h) K(i) h[K(i + 1) K(i)]

for all integral h. As a special case when h = 2

K(i + 2) K(i) > 2 [K(i + 1) K(i)]


K(i + 2) 2 K(i + 1) + K(i) > 0 (3)

Denote by (j)K as the jth difference of K(i). Then

A K = K(i + 1) K(i)

S(2)K = (A(1)K) = K(i + 2) 2 K(i + 1) + K(i)

Notice, however, that this second equation is just equation (3).

Hence, we have proven the following lemrra.

Lemma 1: The second difference of functions with the convex

property is non-negative.

Therefore, we can conclude that

A (K = K(i + 1) K(i) is nondecreasing. (4)

Now consider
A ( [K(i + x) K(i + x y)] =

K(i + x + 1) K(i + x) K(i + x y + 1) + K(i + x y)

But this is non-negative since

K(i + x + 1) K(i + x) K(i + x + 1 y) K(i + x y)

from the property in (4). Hence,

K(i + x) K(i + x y)

is nondecreasing in i for all x and y. Then surely,

D. = E E [K(i + x) K(i + x )
i x y

must be nondecreasing in i and hence crosses A at most once.

This proves the following important theorem.

Theorem 1: For r.=l,10 (i, a) has a solution given by a control

limit rule, i.e., there exists a unique number such that

if and only if with one period remaining there are more

than this number in the system, will a closed server be


For the case when the server is open with one period re-


K1(i, a ) = min K(i) + B + i Pij K(j)

K(i) + + Cat q K(j)
jij IJ i

This leads to observing when

K(i) + B + a I pij K(j) (-) K(i) + C + a qij K(j)


S [pij qij] K(j) () C- _=

The preceding theorem then also applies here, hence guaran-

teeing a unique point at which the server would be closed.

Proof for arbitrary n

For n>l it must be shown that (j*, i*) policies are opti-

mal for,

0 0
n (i, a) min K(i) + a p (j, a)
n j= j n-

K(i) + A + C + a qij n- (j, )


1 0
4 (i, a) = min K(i) + B + a 4 p n- (j, a)
n ij n-1

K(i) + C + a Z Pij (j, a)

Before this is shown, let us prove the following lemma.

Lemma 2: For n=l, i* > j*

Proof: For n=l, i* must satisfy

K(i') + c Y Pi*j K(j) K(i*) + A + C + a E qij K(j) or,

i*j qi*j] K(j) >+C

Similarly j* must satisfy, K(j*) + B + ar pj.j K(j)

< K(j*) + C + Z qjj K(j) or,

S[p qj*j] K(j) C-B

Since A + C > C B and Z [pij qij] K(j) is nondecreas-

ing, i* j* must hold.

Now let us establish the existence of the values (j*, i*)

for n=2. We have that
0 o
02(i, a) = min K(i) + a pi 40 (j, a)
(( 5)
K(i) + A + C + a Z qij l (j, a)


1 0
4 (i, ) = min K(i) + a pij (ja) + B
K(i) + C + a z qi (j a)

Just as before we can write

0 0
Pij 4 (j,a) = E E (i+x,a)
ij 1 x y I


1 1
1 qij l (ja) = E E Vl(i+x-y,a),
0 1
and,as before, if it can be shown that 4l(i+x,a) l(i+x-ya)

is nondecreasing, then so is

0 1
S[Pij ( a) q (j', )]

which implies that the alternatives in (5) and (6) can cross

at most once and the result will be established.

.l(i+x,a) = min K(i+x) + i a i+xj K(j)

K(i+x) + A + C + aZ qi+x,j K(j) (II)

min K(i+x-y) + B + a Z pi+x-,j K(j) (III)

S= (i+x-y,a)
K(i+x-y) + C + i+x-y,j K(j) (IV)

where the (I), (II), (III) and (IV) notationally correspond

to that particular alternative.

It is necessary to consider only the possibilities when

the minima are (I) and (III), (I) and (IV), and (II) and (IV),

i.e. (II) and (III) cannot concurrently be minima by lemma 2.

Now, let us formally prove the following.

0 1
Theorem 2: (i+x,a) (i+x-y,a) is nondecreasing.

Proof: When (I) and (III) are minima 1 (ix,a) (i+x- )=

K(i+x) K(i+x-y) + a Z[i+x,j i+x-y,j] K(j) B

which is nondecreasing, the first two terms from assump-

tion(i)and the second two since K(i+l) K(i) is nonde-


When (I) and (IV) are minima,

0 1
1(i+xa) (i+x-y,a) = K(i+x) K(i+x-y)

+ a1 p q) C
+ aZ [Pi+xj i+x-y,j K(j)- C

which is nondecreasing. This can be shown be replacing

the bracket by

[Pi+xj Pi+x-y,j + Pi+x-y,j qi+x-y,j] K(j)

which is the sum of two nondecreasing functions.

Finally when (II) and (IV) are minima

0 1
4' (i+x) (i+x-y,j) = K(i+x) K(i+x-y)

+ a qi+x,j i+x-y,j] K(j) + A

which is nondecreasing as was the first case considered.

This completes the proof.

This, then,leads to the following:

Corollary 1: There exist numbers (j*, i*) for n=2 such that

(i,a) and 12(i,a) have optimal solutions of the required


Now, let us introduce the hypothesis that,
0 1
for n=3, 4, ... k,4 (i,a) and (i,a) have control limit
n n

rules and, also,

S 0 (j,a) (j- qa)]
Z [Pij 1n-i 1ij n-l '

is nondecreasing.

It becomes necessary to show next that

0 0
+(i,a) = min K(i) + a ~ pij k(j,)

K(i) + a qj q(ja) + A + C


1 0
k+l(i,a) = min K(i) + B + u p ij (j,a)
k+1 k

K(i) + C + a qj .(ja)

have the required property. This is done in precisely the

same manner as before, i.e., it is shown that

0 1
i [Pij -k(jr, ) ij k '( )

is nondecreasing by looking at

0 1
0k(i+x,a) k(i+x-y,a)
k k I
Again, we shall first show

Lemma 3: For each n, i* > j*.

Proof: At i*, the following must be true,
0 1
K(i*) + a i jn-l (j,) > K(i*) + A + C + qijn-l (j)

0 1 A+C
S i(ja) q i n- (j'a)] --
i [i j n- i j n-1 -

At j* the following must hold,
0 (j) K(j*) + C +
K(j*) + B + apXp j'- (j'a) jjK(j*) + C + pj j

Jni 1 C-B
jij n- jj -l(j,) a

Since A + C > C B and the left sides are nondecreasing

by the induction assumption, it naturally follows that

i* > j*.

Theorem 3:

k0(i+x,) = min K(i+x) + a Pi+x,j k-(j,a) (I)
SK(i+x)+ Lx 1

K(i+x) + a q k-(ja) +A + C (II)
i+x,j k-1)

-min K(i+x-y) + B + a i+xy ) (III)

K(i+x-y) + C + ai q., ( 1(j,a) (IV)
i+x-y,j k-l (IV)
(where the (I), (II), (III) and (IV) are unambiguous) is


Proof: From lemma 3 we see once again that it is only neces-

sary to consider when (I) and (III), (I) and (IV), and

(II) and (IV) are minima.

When (I) and (III) are minima the difference is

K(i+x) K(i+x-y) + al [pi j Pi] ) B,
i+x,j i+x-y,j k-1
which is nondecreasing; the summation is nondecreasing

which follows from the discussion on piecewise convex

functions in Appendix I.

When the minima are (I) and (IV) we have

K(i+x) K(i+x-y)

0 1
Sa [Pi+x,j 9k-1(j'a) i+x-y,j k-l(j'a)] C.

This is nondecreasing since

i+x,j -l(j,a) qi+x-y,j k-. (j'a)

Si+x,j Pi+x-y,j] (k-j'c)

0 1
i+x-y,j k -1') qi+x-y,j -1 (j,a)

and the first two terms are nondecreasing by the first

part of this proof and the second two by the induction


Finally, when (II) and (IV) are minima

K(i+x) K(i+x-y) + a [qi+x j-qi+xy,j ]k (jj, ) + A

is nondecreasing by again using the results of the dis-

cussion from piecewise convex functions. This completes

the proof.
0 1
Hence, it is established that +l (j,') and .k+l(j c)

have control limit optimal rules.

Thus, we have established,

Corollary 2: For any n, t 0(i,c) and L (i,a) have solutions
n n
which are control limit rules.

Extension to Other Objectives

The theory behind the next result is due largely to Bellman.2

Theorem 3: A control limit rule is optimal for the infinite

horizon model.
Proof: Let us prove this theorem for 4. (i,a); the other follows


We know that

n (i,a) = mi




Tl 4 ) = K(i)
1 n-1

+ ij n-1 ( )

+ A + C + E qi.. n-l(j,')
:1j n-1

+ Z Ppij n- (j ,o)


T (9n_ )
2 n-1

= K(i) + A + C + a Z qj (ia)


O (i,a) = min
n j=1,2

[T (1 -1
j n-1

Let j(n) = index which gives the min [T. (4n)]

which implies that

0 0( )
n j (n-1) n-1

STj(n) ('n-


o n
4n (ia) = Tj(n) ( 0)
n+1 j (n) n.

S (n-l)
- j(n-1) n


n + (n-i) n 'n- n
ln "I 0. 0 )n T (i^)|
0 0
Tj(n) ( ) Tj(n) (n- )

0 0
< max[IT( 0 n--T ( ]
j=1,2 nj
< max[ K(i)+a Pijcn l(j'a)
)- )n-1

K pi jp 0(j,a)I ,K(i) +
i n

1 1
A+C+a Z qij (j, a)-K(i)-A-C-a Z1 q (j ) ,

< max[la i p n (j,)-a Z p n(j,) ,

< p | (ja)- (j,a)
ij n-l n

max (i,0a max n n+1 (i) 1 Pij n- n '
i i

0 0
n1 n


u = max[ (i,) (ia) ,]
n n n
and hence
2 n
u n+1- n- n-1- 1

and therefore

GO 00 u
S n 1
E u < Z a u -
U n+l- i 1 = -a
n=0 n=O

and we conclude that

0 0
E n (ia) (i, )]
n+1 n

converges uniformly and that 0 (i,a) converges to t (i,a)
for all i, where 0(i,a) satisfies,

0 (i,a) min (i) + aZ p j (j,a)
K(i) + A + C + a Z qij (j,a)

completing the proof.

Now the significance of restricting our decision set to

be finite when the system is closed will become apparent. We

shall assume that for any c>0, an M(c)3: Pr[X > M(t)]
and,henceforth, consider opening only in a finite number of

states. When the server is open, assuming that the arrival

rate is less than the service rate is sufficient to guarantee

that the queue size will remain finite; hence, only a finite

number of decision points can occur.

Finally, we can prove the following:

Theorem 4: A control limit rule is optimal for the averaging

cost criterion.

Proof: Let i be the optimal crossing point of 0 (i,a). Let

{as}, with lim as = 1, be a sequence such that i = i*
S-+= S
for all s. Since there are at most a finite nunmer of

states where the system will be opened, such a sequence

and i* must exist. Let R be any rule which is not con-

trol limit. Then certainly

0 > 0 U

0 0
lim (l-as) R (i,as)> lim (1-as) (i,as
ss sR s

and from Arrow, Karlin and Scarf,

S= lim (1-i ) 0 (i,a ) where
R s Ri s

0 1
R = lim Z g(Xt) and where
T-m t=l

g(Xt ) denotes the cost incurred when observed in state

X Therefore, %. where is the optimal value
STherefore, R -
under the control rule i* and hence i* is the optimal
rule. The proof is identical for .

Closing When There is a Non-empty Queue

Many have conjectured that a server will only be closed

if no customers are awaiting service. Others have said that

the closing point is some function of the system parameters.

Let us answer this question by showing under what conditions

the following theorem is true.

Theorem 5: The server will never be closed when any customers

are present in the system.

Let us assume that we have established a unique point,

say i*, at which the server should be turned on or opened.

Let us also assume, contrary to the theorem, that the optimal

operating policy,n, designates to close the server at some

number j*, 0 < j* < i*.

Since the process is in equilibrium, we know that with

probability one, state j* (or less) will be reached once the

server has been turned on. Hence, the process will alternate

between states j* and i*.

There exists, however, a number k, 0 < k < j*, such that

the expected cost under policy n is equivalent to the expected

cost under policy ni, where,

nI: Open the server when there are i* j* customers in

In fact it will alternate between states less than or
equal to j* and greater than or equal to i*.

line, close when the line is empty and keep k3 customers

on the side at all times.

Define now policy 12 as follows.

H2: Open the server when there are i* j* customers

in line, close when line is empty, and after some finite

time serve the k customers.

Now for each possible objective function consider the

following three policies.

1) Total Expected Undiscounted Cost--Here the additional

expected cost under nl, that of holding the k customers to

the side becomes infinite, while the additional expected cost,

that of serving the k customers, is certainly finite and hence

12 dominates H1 and, therefore, also R.

2) Average Cost--The holding of the k customers under

policy H1 occurs for each period and hence adds a positive

quantity to the average period cost. However, the cost of

serving the k customers occurs for only a finite number of

periods and hence adds nothing to the average period cost.

Hence, again n2 dominates both H and Hn.

3) Total Expected Discounted Cost--Here, both the holding

cost of the k customers along with the cost of serving these

k customers is finite and hence a comparison is necessary.

For a discount factor a the cost of holding the k customers is

at least n K
n K(k)
K(k) E a -

3 If k is not integer keep instead [k] customers off to the
side, so if anything H1 dominates H.

The cost of serving the k customers, which we assume takes

i periods, along with the holding ccst of these customersis

bounded above by

i-1 1-1
K(k) E aU +C E rn
n=0 n=0

= [K(k) + C]
1 -a

To choose 2 over n we must require that

Kk)1 a K(k)
[K(k) + C]
1 a 1 a

2 C
& K(k)+C

Hence, if the discount factor is large (a reasonable assump-

tion), 2 dominates H as well as nR.

4) Finite Horizon-In this case it is possible to close

the server with customers present. This event occurs if the

holding cost is small compared to the operating cost.

Again, the difference between analyzing my system and

that of analyzing the system after each arrival and departure

is that k in the preceding argument is just j*.

Concluding Remarks

Finally we can say for the single server queue that an

optimal stationary policy always has one of the following forms:

a) Always keep the server closed until M is reached.

This case arises when discounting is used and the holding cost

is small compared to the operating cost.


b) Never close the server. This may occur when the

server set-up and shut-down costs are large compared to the

operating cost.

c) There exists a pair (j*, i*) with the interpretation,

open the server only if there are at least i* customers present;

close the server when there are no more than j* customers

present. The important special case of j* = 0 is discussed

in Theorem 5 and that which followed.



Once existence of optimal solutions has been established

it becomes necessary to find an i* which might make it worth-

while to "dismantle" the service station. The closing of the

service station under a (0, i*) doctrine will cause increased

holding, and set-up and shut-down costs. The savings from

the partial elimination of the station, during an idle period,

might not outweigh these increased costs. Hence, an optimal

i* must be found and the cost therein achieved must be com-

pared to that of a simple queue, i.e., never close the server.

A simple computational scheme is developed to make this com-


The second part of this chapter makes use of Howard's

approach in analyzing the system from a computational view-

point. Description of the algorithm, calculation of probabil-

ities,along with some interesting results are presented.

Averaging Computational Scheme

Calculation of Average Queue Length

Let us consider an averaging cost criterion, i.e., find-

ing the optimal policy for the average cost per cycle. By a

cycle we mean the following:

Let us begin at the moment our service station has been

closed dc::n there are no customers waiting for service.

Call phase 1 that time it takes for i customers to accumulate

before the service station. By the second phase, is meant

the time from when the ith arrival enters the system until

the end of the present time period. Let the average number

of arrivals in this phase be denoted by a'. Let the third

phase consist of servicing these i + a'customers along with

their generation arrivals (all arrivals occurring while the

original i + A'are being served) until there is no queue and

our service station is once again closed down. Phases I, II

and III make up one cycle.

The average time for phase I is i / k, the assembly time

for i customers.

Let F(t) be the distribution of the length of time for

phase II. Let E(t) be the expected length of time for phase II.


\ i t (t)j
ja= je' e dF(t) = t dF(t) = \E(t)
0 j=0 J'

and hence E(t) =
x (1)

be the traffic intensity. In addition define P, and P as
the time of bus- and clear periods for ordinary queuing sys-

tems. Then, because of the Poisson arrival process,

P = 1/\

Also, the fraction of time the server is busy is given by

P +P
b c (3)

On combining (1), (2) and (3), we obtain,

P 1
b P-,

This is the expected length of the busy period for one cus-

tomer and, hence, the expected duration of the third phase

is (i+&')/(u-.).

The average length of a cycle can then be easily obtained

by summing,

i (i+ ') i (P-.)+a' (p-A)+(i+C')A
T + +
A ia-A A (v-\)

u (i+L') i + C'
.\ (1-.\) X (l-p ) (4)

Also, the fraction of time the system is in each of the phases


i i (1-0)
AT i + cJ (5)

CL C'(1-c)
AT i + C' (6)

i + (7)
(p-A)T -
for phases I, II, and III respectively.

Let us adopt the following notation:

Pkj = Pr There are k customers in the system and there

are j available servers

K = 0, 1, ... and j = 0, 1

tt-A random variable denoting the length of time from the i

arrival to the end of the next observation point.

T A random variable denoting the time between the

arrival of a customer in the third phase until the com-

pletion of the particular service in progress.

T A random variable denoting the time between the

arrival of a customer in the second phase until the

end of the second phase.

W Average queuing time

L Average queue length

Let us note some important relationships:

Since phase I is just the arrival of customers through

a Poisson stream all states in phase I will occur with the

same probability. Hence, from (5)

P -0 K = 0, 1, ..., i-i (8)
KO i+ '

Also, from (6) the probability of being in phase II is

= _(lp
.PKO J +. '
K=i (9)

Finally, the associated probability for phase III is,

SpK1 = (10)

Next, Cox and Smith4 show that the forward delay, T, is

distributed so that

E(T) = E(t)(
2 (11)

where t is the original random variable and v, the coeffi-

cient of variation of t. For the exponential distribution,

v = 1.

Now, we proceed to analyze the queuing time of customers

arriving in different phases. For a customer who arrives in

phase I, where there are K customers present, his waiting

time is,

(i-l-K)/\+ E(tt) + (K+1)/p

.An arrival in the second phase when the system is in state K

can expect to :wait,

E(t ) + (K+l)/w

Finally, a customer in the third phase, when K are present

at his arrival, will wait on the average,

E(T ) + K/u

Now we are in the position to write down the average

queuing time W:

W = Z pK [(i-1-K)/\ + E(tt) + (K+1)/p)]

+ L pKO [E(Tt) + (K+1)/u]

+ Z Pj [E(T ) + K/p]
K=l J s

Insertion of (1), (8), (9), (10), and (11) yields,

= 1 i(i-1) (1-c')

S== [(L+1) +
A 2(i+a')

'i( 1- ) ~2a (1-P) ( 1+-.
+ +
i+a' i+a' 2

From Little's formula,

L = .W

the average queue length is then,

2 i+v
P i(i-l) ai
L = p + + + [i+ar (---)]
1-0 2(i+a) i+a' 2

which is the average queue length of a simple queue, given

by the first two terms (Pollczek-Khintchine), plus the addi-

tional queue length due to the intermittent operation of the



Now it is possible to write down the cost function for

the averaging criterion.

First, in one cycle a cost will be incurred due to
set-up and shut-down of the server. Also we can expect a con-

tribution of K(L) as the average holding cost per cycle.

Finally, in one cycle there is savings due to the decreased

use of the server. The server is being used only when in

phase III and, hence, the cost incurred from its use is expected

to be oC/T.

Therefore, the average cycle cost is

Ct(i) = K(L) + (l- A+B), C/T

To find the optimal opening point, i, we may differenti-

ate Ct with respect to i, set equal to zero, and solve for i

(an integral value is naturally required). However, this is

impossible as given, as it is necessary for the form of K(-)

to be explicitly given.

A Specific Case

First, for the sake of example cnly, and second, to com-

pare the results with some previously cited works, let us

assume that the holding costs, K(-), are linear with respect

to the averaging values. Then,

2 lt1+v
D i (i-l) O' t
(i) = K [o+ + + [i+a'( )]]
t i) 2 (i+a') 2

+ (A+B)(1l-p)
+ PC/T


1 Ki
SK. (i+.') (i-1)- (i-1)
C 2 2
ai (i+ ') 2

+ K.(i++') L[Kai + Ka'2 t (A+)

Ki2 + 2K + K'-- 2 (A+B) (1-,A+) 0
(i+, 2 (i+d)


Ki + 21KP + K,'['-1-v2) 2(A+B) (l-p) = 0

t K

which has roots,

( -2t' 4a2- 4 [ (,_-1- t2) 2 (A+B), (1- )] )/2
t -K


S \/,,2 [,( ,- vt2 2 (A+B) A (1-C )
St K

Simplifying we obtain,

2(l-P) (A+B) + 2'(1
K t

The positive root yields the minimum and, hence

i = -'+ 2(1)(+B a(l+a2) (12)

The optimal solution is then obtained by comparing the

cost of the nearest positive integers [i], [i+1] and the pos-

sibility of permanent server availability. This latter cost

is just

C = K [p+ -] + C/T
s 1-p

That is, i* is defined by

C(i*) = min [C[i], C([i+l]), C ]

(in the last case i* = 0).

It is interesting to note that the value of i obtained

by differentiation is not a function of the running cost, C,

of the server. Actually, this result is not surprising as

over a reasonable length of time the same number of customers

will be served on the average; hence, the server will be running

for the same length of time.

Finally, if the system is observed after each departure,

rather than at equal intervals, phase II disappears. Therefore,

(12) becomes

S(A+B) \ (1-c)

This result agrees precisely with that of Heyman's.

Computational Experience Using Ecward's Algorithm

Brief Description of Algorithm

Given a process which recieves a reward qi at the begin-

ning of a period when found in state i, then, if Vi (n) is

the present value of the total expected return when in state

i with n transitions remaining, we obtain

V.(n) = q. + Z [pij V (n-l)]

for each i and n where a is some discount factor and a total

of M states.
With the introduction of alternatives, Howard has devel-

oped an algorithm which minimizes these present values for

an infinite horizon problem, i.e., replacing Vi = lim V (n)
for V.(n). This routine can be summarized by the following

flowchart in Figure 2.

The iteration process can be entered in either box and

Howard guarantees that each cycle will yield a present value

at least as good as the one previously obtained. He also

guarantees that there is convergence to an optimal policy

and this is noted by having policies on two successive itera-

tions identical.

In the problem at hand either of two decisions may be

made in any of the 2:1 states. The states are characterized

[1] Veinott 16 has guaranteed convergence in a finite number of

Figure 2: Policy Interation with Discounting

by the number in the system 0, 1, ..., M-1 and whether the

server is closed or opened. The cycle was entered in the

policy improvement routine by choosing all the present values

as zero. The first step, therefore, was minimizing immediate

returns, i.e., choose the alternative that chooses

q. = min K(i)
K(i) + A + C

when the server is closed and

K(i) + B
q. = min
K(i) + C

when the server is opened. The new present values were then

calculated and iteration process continued until convergence

was obtained.

Calculation of Probabilities

In order to make use of the algorithm explicit values

for the transition probabilities,pij and qij had to be known.

When the server was closed, p.., the probability of going

from i to j in one time period,was simply the probability of

j-i arrivals, where the arrivals are Poisson distributed with

parameter .\

With the server opened, the probabilities qi are derived
as follows. Prabhu finds the Pr (there are
at time t I there are i in the system at time 0)

= Pr(Q(t) 1 j Q(0) = i)

e-(e+u)t ab ta+b e-(\+p1)t a+j+l b-j-l ta+b
A j-i) aab! (-j-i-2) a!b'

where A(j) denotes the set of integers a, b such that a>0,

b>0 and a-b
Subtracting from this quantity

Pr (Q(t) < j-1 I Q(0) = i)

we get

Pr (Q(t) = j I Q(0) = i)

Next, setting t=l, i.e., one time period, we establish

q eha b a b a+j+l b-j-1
qi = e (+ LL + Z i- }
ij a!b! abi) a a!b!
1 a-b=j-i i-1) a A(-j-i-2)

Since we always make the decision to open at state *1 and

above, these probabilities are truncated at M!. This is neces-

sary as Howard's routine allows for only a finite number cf



A FORTRAN program was used to calculate these probLailities

and then incorporated into another which was Howard's policy

improvement routine. Some of the results obtained were both

reassuring and enlightening. Since, however, sensitivity on

one parameter was accomplished while the other parameters were

fixed at particular levels any definite conclusions were dif-

ficult to reach.

Sensitivity to discount factor

Theorem 5 in Chapter II stated that it was possible to

close the server with customers present for the infinite



FOR i=1.0, u=2.0,
A=10, B=10 AND C=40

j* for K(X) =

a : X2 + X e"

0.9 0 0 0

0.8 0 0 0

0.7 M 0 0

0.6 M 0 0

0.5 M 2 0

0.4 M 4 1

0.3 M M 2

0.2 M M 3

0.1 M M 4

horizon discounted case. For fixed costs, however, this was

only true for small a. Table 1 gives j*, the closing point

as a function of three different parameter values. Notice

that the values are not only small but also as the holding

costs became "more convex" the value of the discount factor

when the server is closed with customers present decreases.

Other results

Certainly extreme cases led to extreme results as far

as determination of i* and j*. For example, for high set-up

and shut-down costs and low operating costs as soon as we open

we always remain open. Also, if the holding cost is small

compared to the fixed costs once we close, we will always

remain closed (since the fixed costs are constants, this only

occurred in our investigation when the holding cost was linear).

An interesting observation was that the sensitivity of i*

to the ratio of the arrival and service rates (A/,) was very

slight. However, i* was quite sensitive to the magnitude of

X and p. When A and p were chosen large (>1) i* turned out

to be only 0 or 1. However, when their magnitudes were decreased

(A=0.1, p=0.2) i* had the value 4. Certainly there is varia-

tion here depending on costs but large arrival and service

rates imply larger shifts in state. This means having large

holding costs and soon after no holding costs and conversely.

This implies many set-up and shut-downs and hence i* is very

low so that the server would only be shut down when empty.




Let us return now to the more general problem, that of

finding optimal policies for the operation of a service mecha-

nism with an arbitrary number of servers.

The assumptions made at the beginning of Chapter II will

still hold, except the following change:

(I') Assume there are s servers. Their service times

are independent exponential random variables each with

rate p. Customers are still processed in the order of

their arrival. If a server is closed down while proc-

essing a customer, that customer rejoins the queue.

Also, just as in the single server case, we shall assume

that for k, the number of open servers, our arrival rate is

such, that given any E->0, there exists and M(E) such that:

P[Xt>M(E) ]<

This along with the assumption that s.\
decisions will only have to be made in a finite number of states.

The cost structure is affected only in that there is now

an operating cost, set-up cost and shut-down cost for each

server, respectively C, A, and B. The transition probabilities

will now be denoted as

i = PrXt+l = j = i and k(=0,l,..., s)servers are opened}

where Xt is again the number of customers in the system at

time t.

Form of the Optimal Policy

In this section we will describe a policy which we will

define to be optimal. This policy most closely resembles that

of the (j*, i*) for the single server.

Given the state of the system,we are interested in deter-

mining unique numbers which tell us how many servers will be

open after a decision is made. Notationally, let us describe

an optimal policy as follows:

(i,k : j J ,..., k-1; ik+l '' is) (1)

where the i, k represents the present state of the system,

i.e., i customers present and k servers opened. The sequences
(j } and {i} are nondecreasing sequences with the following
If i m-1- m -
after decision). If j _lim, close k-m servers
(have m open after the decision). If i>i or i
all the servers open or closed respectively. If i does

not meet any of these conditions keep k servers open.1

[ Note how this conforms to the original form for an optimal
policy described in Chapter I.

Figure 3: Costs associated with a multi-server queue.

An optimal policy may permit going from j to j+l or j to

j-1 for all j. If this is the case both sequences are increas-

ing. However, an optimal policy may dictate a state where

the decision to open only one server will never be made. In
this case i+ = iz (or, possibly j- = jj. This is best

illustrated by a simple example.

Let us assume that s=3 and presently i=0. Figure 3 depicts

two possibilities as to what the cost might be as a function

of the number of servers to be opened.
In (a), il, i2 and i3 are well defined to be the crossing

points of the (0,1), (1,2) and (2,3) intersections respectively;

and the sequence of i's is increasing. In (b) it is never opti-

mal to have one server open so in order to conform to our rules
our optimal policy will have i = i2 and the sequence of i's

is nondecreasing.

Dynamic Programming Formulation

Now that the elements of the model are given let us again

set up the dynamic programming formulation for the finite hori-


Define n(i,a,k) as the minimum expected cost with i

customers present, k servers opened with n periods remaining

in our horizon,using discount factor a.

Then our recursive relationships are:

K(i)+kB +a p (j,a,0)

K(i)+(k-1)B+C+a E P jn0(j,,1
ij n-i

(i,a,k) = min K(i)+kC +a E p i4 n (j, ,k) (2)

K(i)+ (k+l)C+A+a 1 pij .+-I (j ,a,k+l)
ij n-1

K(i)+sC+(s-k)A+a ,p p (jcs)
ij n-l


0 (i,a,k) = K(i)

and as an analogous assumption from the single channel system

we can say that there exist numbers M, M, ... Ms which

demand that

(m,a,k)=K(m)+(k+l)C+A+a L p (ja,k+l) for mLMk (3)
n -m] nj-1 k

where there is a one-to-one correspondence between the k and

Mk Since this sequence (Mk} is finite there is certainly a

maximum and for simplicity we will impose (3) for m > max {MIk}=!.

All this says that whenever M customers are reached and k ser-

vers are opened we will automatically open one more server.

Furthermore, the assumptions that K(i) is nondecreasing

and convex will again be made, along with assuming that for

each nondecreasing function h(j)

SP.k h(j)

is nondecreasing in i for each k.

Proof for Finite Horizon

The same sequence of developing is used for the multi-

server system as was used for single server. In this chapter

only this logical sequence of steps will be mentioned to ensure

our results for an arbitrary number of servers. Proofs for

the theorems will appear in Appendix 2.

For n=l, let us illustrate how the result carries by

considering s=2. Then,

1 (i,c,0) = min

#l(i,c,l) = min

l,(i,a,2) = min

when the server

E K(j)[p

and that

K(i) +C K(j)

SK(i)+A+C +L. E p1 K(j)

K(i)+2(A+C)+cE p i K(j)

K(i) + B C+ E p K(j)

SK(i)+C+a 2 pi- K(j)

K(i)+A+2C+a p 2 K(j)

K(i) + 2B + a pi K(j)

K(i)+B+C+a C pj K(j)

K(i)+2C +a p K(j)

is closed it must be shown that

- p .] is nondecreasing

SK(j)[ 2
(j). Pij] is nondecreasing
13] 1j

(The sum of these two is just

0 2
E K(j)[p p. ]
ij 13j

which would certainly be nondecreasing as it is the sum of

two nondecreasing functions.) Doing this leads to proving

Theorem 1: For n=l the policy of form (1) is optimal.

To extend the results for arbitrary n, let us point out

the steps that make the analysis for the single server and

that for an arbitrary number of servers similar.

For n=2 it becomes necessary to show that

kl k2
U1(i+x F y.i,,kl) 1(i+x E yi ,,k2
i=0 i=0

for each kl and k2,such that k2 >kl, is a nondecreasing func-

tion of i. This is equivalent to saying that there is at most

one crossing point for any two alternatives and since the curves

are nondecreasing (1) will be ensured optimal. The proof for

this is the proof of Theorem 2 in the appendix and is analagous

to Theorem 2 of Chapter II.

Again, continuing with the induction hypothesis, i.e.,

assuming n (i,a,k), n=3,4,..., k has an optimal rule given

by (1), the enumeration procedure becomes more difficult but

once again Theorem 2 of Chapter II follows in precisely the

same manner (this once again is so because alternatives only

have to be compared two at a time). This is done first, by
providing the analog of Lemrma 3 (proving ik+1l k- ) and

secondly, by using the transitive relations that if a-b is

nondecreasing and b-c is nondecreasing, then a-c is nondecreas-

ing (a-c=(a-b)+(b-c)). The details are in Appendix II.

Also, the discussion on piecewise convexity is simply

extended. The definition allows a function to be the smallest

of a denumerable number of convex functions and, hence, the

addition of the extra alternatives still permits n(i,a,k)

to be piecewise convex. Also, the argument related to the

terminal points holds since having the possibility of s

servers just allows for the possibility of s different weights.

This completes the discussion for the finite horizon and

we have shown that

Theorem 2: For an arbitrary number of servers our optimal policy

for the cost functions given in (2) is satisfied by (1).

Infinite Horizon and Averaging

Proceeding as before we shall prove that (1) is the opti-

mal policy for the infinite horizon, i.e., when

(i) + BK +a p p (j. ,)

K(i)+B(k-l)+C+:a 1 p $(j,a,l)
k .
*(i,a,k)= K(i) + kC +a p j (j,a,k) (4)

K(i) + (k+l)C+A+a L p k 1 (j ,ak+l)

K(i) + sC + (s-k)A+cr Z Pp (j,,s)
Again the proof will be done for only one state (i,k) but by

its nature is seen to hold for all states. The proof is

Theorem 3 in Appendix II.

Moving into the discussion on the averaging criterion,

the assumption that for each number of open servers there are

only a finite number of decision points we are able to prove

that (1) is the optical policy as shown in Theorem 4 from

Chapter II. Theorem 4 in Appendix II gives this proof implying

that (1) is the optimal rule for t(i,k) where

T(i,k) = lim (l-a )4(i,as,k) where

1 T
W(i,k) = lim i Z g(i,k) and g(:t,k)is the one step loss
T-o t=l

Concluding Remarks

Certainly, the extension of Theorem 5 in Chapter 2 holds

for the multi-server model. That is, we will never close all

the servers with customers present, except possibly when there

is a finite horizon.

In essence, therefore, this chapter along with Appendix II

shows that the multi-server and single server model are very

similar, in the sense of choosing optimal policies. This is,

however, an exception rather than the rule in dynamic program-

ming formulations. Bellman points out that under very general

conditions when there are only two possible decisions that

there is some point which uniquely separates our decision region.

However, for more than two decisions he maintains that in general

our decision regions are not so divided.[2]

Bellman, R., op.cit., page 75.



The purpose of this paper was to ascertain optimal poli-

cies for queuing systems. The form of the optimal policy for

the single server had been found before, but only when obser-

vations were at the end of arrival and departure epochs, rather

than having observations in equal time intervals. Also, the

only work previously done in existence of optimal policies

for multi-server models was a two-server case discussed by

Heyman. In this discussion, he always had one server open

and the policy was to determine when to open the other. We

allow for any finite number of servers. We are able to con-

clude that the number of servers open at some point in time

is given by an intuitively appealing rule. This rule is

analagous to the (s,S) policy in inventory theory.

The applications of the work in this paper are quite wide-

spread. In any situation where there is a variable number

of service stations which may be open, one would know how many

servers should be opened or closed. More might be learned

by observing additional computational results, as the purpose

of this paper is not to find optimal policies but only the

form at which they take on. Sobel points out many specific

problems relating optimal policies for queuing systems to

inventory and replacement theory.

There are several extensions to be seen from the work

done in this paper. Certainly there is a need for greater

computational facility in obtaining the optimal policy for

a given model. At present the closest computational experi-

ence for the multi-server model is in a paper by Moder and

Phillips.2 Also, there is no reason why Howard's scheme

cannot be extended to deal with this case. Computational

experience is very important for a problem of this type as

its application will only become meaningful when the param-

eters which we might be free to choose are set at the proper

levels. That is to say that even though our optimal policy

is determined only from the number in the system and the

number of servers open, there may be other parameters, such

as service rate, which might substantially affect our optimal


The main restriction that had to be made, so that periodic

review can be used, was that our arrival and service distribu-

tions were exponential. As was mentioned this assumption could

be dropped if we are willing to increase our state space.

That is, the state would be not just a number in the system

and the number of open servers but also the time from the last

arrival, if arrivals were to become general. Investigation of

this problem is to be considered. Computationally, one would

almost certainly have to use Jewell's Markov-renewal technique.

Another extension is to consider finding optimal policies

when there are several machines with different service rates.

This was avoided because one must either make a predetermined

rule, or include the finding of the rule as part of finding '

the optimal policy to determine which of the available machines

an arriving customer will be serviced on. When this is done

one can introduce switching type policies as alternatives,

i.e., if a server is closed down while serving a customer,

the customer may immediately switch to another machine. Heyman

discusses rules of this type.

A slightly different but very interesting problem can be

thought of as an extension is that of finding the optimal

policy as a function of some other parameter. In particular,

finding an optimal rule as a function of the arrival rate,

rather than of the number in the system, may have useful appli-

cations. If one could check on the rate at which customers

entered a supermarket, for example, it might be possible to

predict when servers should be opened or closed so as to avcid

idle time and long queues.

Of course, balking or reneging can also be introduced to

the system. In connection with this we could consider a modi-

fied loss system which has application to telephone switchboards.

That is, consider a customer who tries to enter the system and

is turned away (placing a call and finding all circuits busy)

but will return with a certain probability. Associating a

cost with a lost customer it is to be determined how many ser-

vers should be kept ooen so as to minimize cost.

Another change in the model would be to change the queue

discipline; introduction of priorities may certainly make this

work applicable to time-shared cor. puter systems.



The following is due to Zangwill

Definition: Piecewise convex function (PCF) -- A function

f(Z) is said to be piecewise convex if there is a

sequence of convex functions {gi(Z)} such that

f(Z) = inf {gi(Z)

Definition: Let f(Z) = inf {g.(Z)} be PCF. Define a basic
set B. as

Bi = {ZI f(Z) = gi(Z)}

Definition: Let the terminal set T be defined as

T = {ZI f(Z) = gi(Z) = gk(Z) for i / k}

The next lemma is an important one in the concept of PCF.

Lemma 1: Let {fi(Z)} be a sequence of PCF each with respec-

tive terminal set T.. Then F(Z) = w.f. (Z), with

w. > 0 is PCF with terminal set T = U T..

Proof: The proof is by induction on the number of functions.

For one,the result is clearly true. Assume it is true

for k, i.e.,
k k k
Fk(Z) = w.f. (Z) is PCF with terminal set T = U T.
k I 1 i=l


By the definition of PCF
Fk(Z) = inf {g (Z)}

for some sequence of convex functions.
Let w k+if (Z) = inf{g (Z)} for a sequence of convex

functions. Then

1 2
Fk+ (Z)=FkZ) k+ f (Z)= inf{g (Z)}+inf{g (Z)
i j

1 2
= inf{gi (Z) + g (Z)}
i j
1 2
But gi (Z) + gj (Z) is convex for all i and j and hence

F k+(Z) is PCF.

Now, a point h is in the terminal set of gl () + g () if
1 2 2 1
and only if g (h) + g (h) = g (h)+g2 (h)=g. (h) and
1 g 12
either (a) il/i2 and jl=j2, (b) il=i2 and ji/j2, or

(c) i 2 and j.2. If (a) occurs then h e Tk, if (b)
1 ii and jdJ2.
occurs then h Tk+1 and if (c) occurs then h E T kTk+l.

Also, if h E T then either (a) or (c) occurs; if h e Tk+l
then either (b) or (c) occurs and h e T Another

useful lemma is,

Lemma 2: If g(Z) and h(Z) are PCF then f(Z) = min [g(Z),h(Z)]

is PCF.

Proof: By definition of PCF

g(Z) = inf g. (Z) } and h(Z) = inf {h. (Z)}
i 1j

for some sequence g and h.


f(Z) = min

[inf (gi(Z)}, inf {h. (Z)}]
i j J

= min [inf (g.(Z) hj (Z)}]
ij J

= inf

(gi(Z) h (Z) }

and f(Z) is PCF.

Lemma 3:



E q..K(j) are convex.

Proof: (Two ways are

illustrated -- one for p.ijK(j) and

the other for F qijK(j))

E PijK(j) = E K(i+x)
j=0 'L x

= Pr (x arrivals in time period) K(i+x)
which is a linear combination of convex functions and

hence is convex.

Or as an alternative,

6K(i1+x-y)+(l-e)K(i2+x-y)K(eil+ (1-6)i 2+x-y)

6E E K(i +x-y)+(l-9)E.E K (i +x-y)::E E F(i +(i-)i +x-y)
x y X Y 2 X y 1 2

and hence

Sqij K(j) = E E K(i+x-y)
j=0 x

is convex.

Recalling that,

9 (i,a) = min

K(i) + a Z p._ K(j)

K(i) + A + C + a q K(j)
j=0 ij

and using the definition of PCF we conclude that 4, is PCF.

Next, since

0 0 0
pi l (j,a) = (i+x, ) = E Pr(x arrivals) i (i+x,a)


E q
S~ 1 (j,a) = E E (i+x-y,a)

= I Pr(x arrivals) Pr(y services)41 (i+x-y,a)

along with Lemma 1 enables us to say that I pi 4. (j,a) and

Z qij 0(j,a) are PCF. Since the addition of a convex function

to a piecewise convex function is obviously piecewise convex,

2 (i,a) is a PCF, a result achieved from Lemma 2.

Finally, alternate use of Lemmas 1 and 2 leads to the


Lemma 4: 0 (i,a) for all n > 1 is PCF.
The result also follows immediately for C (i,a).
Clearly, since the set of terminal points does not depend

on the weights,

O 0
SPi+xj -1 (ja) and pi+x-yj k-1 (ja)
k 1 i+x-y kj ki

have the same terminal points. Also, each interval between any

two terminal points of a piecewise convex function is convex.

Hence, to show
[0 P0j 0a
[Pi+xj -1(j') Pi+x-y,j 'k-1 a)]

is nondecreasing between terminal points one can resort to the

discussion used to show the same thing for

E [pij qij] K(j).


Repeated use of the fact that the terminal points are

the same and that the expression is nondecreasing between

these points enables us to conclude,
Lemma 5: Z [pi+x, P i+x-,j] tk-i (j,a) is nondecreasing

in i. Host certainly the result also holds for

Sqi+x,j qix--y,j k-1 (ja)



Lemma: Z [p pi K(j) is nondecreasing for m j=0

Proof: Just as q. K(j) =

E j K(j) = E E ...E
3ij x yl


E E K(i+x-y),
x y

K(i+x-- yj)
m j=l j

Sy. = Y from Chapter II we know that

K(i+x-Y ) K(i+x-y ) for Y >Ym is nondecreasing

in i. Hence, so would the difference in their expecta-

tions be and

E [Pi
j=0 i

Pij] K(j)

for all m
Theorem 1: For n=l the optimal policy is of form (1) in Chap-

ter IV.

Proof: Let k be the number of open servers.


K(i)+Bk + P p. K(j)

K(i)+B(k-l)+ C+c p. K(j)

p1(i,k,a) = min K(i)+ kC+a T p.k K(j)

K(i)+A +(k+l)C+ac p. K(j)

K(i)+A(s-k)+ CL E p s K(j)
1 -

The policy (1) implies that there exist unique numbers

which determines exactly how many servers should be open

with i customers in the system, given k were open. Now,

for any two alternatives m and n, m
K(i) + (Constant) + a L p. K(j) (1)

K(i) + (Constant)' +c ~ Pi K(j)

m n
E [p p j] K(j) = Constant

can occur for at most one value of i. Since the left

side is nondecreasing from the preceding lemma, once

alternative n is chosen, alternative m can never be

chosen again for all nm
nondecreasing, which determines when an alternative should

be chosen.
kI k
1 2
Lemma: [Pij, (jc,kl) Pi t (j,',k2) is nondecreasing

for kl

Proof: This can be written as

E[1l(i+x-Ykla,kl ) l (i+x-Yk2 ,,k2)]

where the expectation is taken over random variables x,

Y1 Yk2. If the bracket is nondecreasing for all i then

certainly the expectation will be. So let us consider

Sl(i+x-Ykl ,a,kl) ~(i+x-Yk2 a,k2)

Now for any i,

i (i+x-Ykl ,,k 0 = K(i+x-Ykl) + (Constant)
1 1 k 1

+a L Pi+x_y ,K(j)

The difference is nondecreasing in i since K(i+x-Yl) -

K(i+x-Yk ) is nondecreasing and the difference of the
summations is by the first lemma in the appendix.

Assuming that n (i,a,k) n=2,3,4,...r has an optimal rule

given by (1) and that

k k
S[Pij n (j','kl) Pi n (j,'',k 0] for kl < k2

is nondecreasing in i, (1) must be shown optimal for n=r+l.

Theorem 2: (1) is optimal for r

Proof: Since

K(i)+kB +a p U i tr(i,a,0)
ij *r

4 + (i,a,k) = K(i)+ kC+ac pi k (i,a,k)
r+L ij r

K(i)+(s-k)A+sC+. Z ps (ias)
I j r

for optimal solution to be (1) it must again be true

for alternatives m and n, m < n, the difference must be

nondecreasing. This will guarantee that the sequences

(j*} and (i*} will be nondecreasing. Since this difference


[Pm 4 (iA,m) pinj (i,a,n)]+(constant)
ij r ij r
it is nondecreasing by the induction assumption.

Theorem 3: 'i(i,a,k), as defined by (4) in Chapter IV, has an

optimal policy given by (1) in Chapter IV.

Proof: (Done for an arbitrary state (i,k))

Define T. (_l ) as the cost associated with the jth alterna-

tive j=0,l,...,s. For example,

T0 0n) =n K(i) + Bk +au pPij 'tn-(ja0)


4 (i,a,k) = min T. ( n- )
n j=O ,. n-l

Let j(n) = index which gives the minimum of T. (n ), j=0,...,s.

This implies that

n (i,a,k) = T j(n-l) (' n- ) < Tj ((n-1)


'p (i,a,k) = T. (' ) < T. (' )
n+1(i,,k) = Tj(n) ) n T- j(n-l) n)


T (i,'a,k)- n (i,a,k)| max [ Tj ( )-T (n( ) ,n- '

ITj (n) n)-Tj (n) (n- ]


j=0 . .,


(K(i)+kB+a E p.. 9 (j,a,0)) ,

K(i)+Ck+a pZ k _(j,(a,k) -
IK(i)+Ck+a Z pi n-
(K(i)+Ck+a E 4P (j,a,k)) ...
ij n

IK(i)+sC+(s-k)A+a 7 pi n- (jas) -
ij n-1

(K(i)+sC+(s-k)A+a 7 pi s (j,cs))l ]

< aZ p k j (j,a,k) ; (j,a,k)l
S ijj nn- n


max 4I (i,a,k) ( (i, a,k) <

aEp maxi (i,a,k) ,
ij i n-1

S n- n

I 1
Define, U n max [ .;.n (i,a,k) .

and hence

U < aU < a U <
n+1- n n-i


n U1
U < aU =
=0 n+l- 1 i-a
n=0 n=0

we conclude that

z [ n+l (i,a,k) p (i,a,k) ]

(i a ,k)


n-1(i,a ,k)


converges uniformly and that

n (i,a,k) a(i,a,k)
where (i,a,k) has been previously defined.

Theorem 4: (1) of Chapter IV is optimal for the averaging

cost criterion.

Proof: Let (jO' jl' '" jk-l' ik+l' is) Pa

be the optimal policy for Y (i,a,k). Let fa t, with

lim a =1 be a sequence such that P = P for all t.
St a

Since the number of possible policies is finite such a

sequence and, therefore, P* must exist. Let P be any

other policy. Then certainly

p(i,(i,atk)> ,(iatlk)


lim (1-at) (i,at,k) > lim (1-at) 4(i,at,k)
toC t-O

and again using the result that

(i,k) = lim (1-at) 4 (i, t,k)

where a(i,k) has been previously defined, it is the mini-

mum averaging cost.


1. Arrow, Karlin and Scarf, Studies in Mathematical Theory
of Inventory and Production, Stanford University Press,

2. Bellman, R., Dynamic Programming, Princeton University
Press, 1957.

3. Blackwell, D., "Discounted Dynamic Programming," Annals
of Mathematical Statistics 36, 1965, 226-235.

4. Cox and Smith, "On the Superposition of Renewal Processes,"
Biometrika 41, 1954.

5. Derman, C., "On Optimal Replacement Rules When Changes
of State are Markovian," Mathematical Ootimization Tech-
niques, University of California Press, 1963, 201-210.

6. Edie, L., "Traffic Delays at Toll Booths," Operations
Research 2, 1954, 107-138.

7. Hadley and Whitin, Analysis of Inventory Systems, Prentice-
Hall, 1963.

8. Heyman, D. P., "Optimal Operating Policies for Stochastic
Service Systems," University of California, 1966.

9. Howard, R., Cvnamic Programming and Markov Processes, MIT
Press, 1960.

10. Jewell, W., "Markov Renewal Programming I and II, "Opera-
tions Research II, 1963, 938-971.

11. Magazine, M., "Markovian Sequential Decision Processes,"
THEMIS Working Paper, 1968.

12. Moder, J. J. and C. R. Phillips, "Queuing with Fixed and
Variable Channels," Operations Research 10, 1962, 218-231.

13. Nemhauser, G., Introduction to Dynamic Programming, J. Wiley,

14. Prabhu, N. U., Queues and Inventories, J. Wiley, 1965.

15. Sobel, M., "Optimal Average-Cost Policy for a Queue with
Start-up and Shut-down Costs," Operations Research 17,
1969, 145-162.

16. Veinott, A., "On Finding Optimal Policies in Discrete
Dynamic Programming With No Discounting," The Annals
of Mathematical Statistics 37, 1966, 1284-1294.

17. Yadin and Naor, "Queueing Systems with a Removable Ser-
vice Station," Operations Research Quarterly 14, 1964,

18. Zangwill, W., "The Piecewise Concave Function", Manage-
ment Science 13, 1967, 900-912.

19. Little, J. D. C., "A Proof of the Queueing Formula:
L = XW", Operations Research 9, 1961, 383-387.


Michael Jay Magazine was born in New York City on April 29,

1943. He was educated in New York elementary schools and grad-

uated from Forest Hills High School in June, 1960. In June,

1964, he received the Bachelor of Science degree in Mathemat-

ics from the City College of New York. In January, 1966, he

received the degree of Master of Science in Operations Research

from New York University. While at New York University he

worked as a graduate assistant in the Department of Industrial

Engineering and Operations Research. Since September, 1966,

Michael Magazine has been at the University of Florida pursuing

the degree of Doctor of Philosophy. During this time he worked

first as a research assistant and then as a research associate

doing research and teaching courses in the area of Operations

Research. In June, 1969, he was awarded the Master of Engi-

neering degree from the University of Florida.

Michael LMagazine is a member of the Operations Research

Society of America, The Institute of Management Sciences, as

well as the honorary societies Alpha Pi Mu and Sigma Xi. He

is married to the former Joan Nemhauser and has one son Rcger


This dissertation was prepared under the direction of

the chairman of the candidate's supervisory committee and has

been approved by all members of that committee. It was sub-

mitted to the Dean of the College of Engineering and to the

Graduate Council, and was approved as partial fulfillment of

the requirements for the degree of Doctor of Philosophy.

August, 1969

Dean, College of Engineeri

Dean, Graduate School

Supervis r Committee:

h harrman

-_ /


University of Florida Home Page
© 2004 - 2010 University of Florida George A. Smathers Libraries.
All rights reserved.

Acceptable Use, Copyright, and Disclaimer Statement
Last updated October 10, 2010 - - mvs