
Full Citation 
Material Information 

Title: 
Optimal policies for queuing systems with periodic review 

Physical Description: 
Book 

Language: 
English 

Creator: 
Magazine, Michael Jay, 1943 

Copyright Date: 
1969 
Subjects 

Genre: 
bibliography ( marcgt ) nonfiction ( marcgt ) 
Record Information 

Bibliographic ID: 
UF00097765 

Volume ID: 
VID00001 

Source Institution: 
University of Florida 

Holding Location: 
University of Florida 

Rights Management: 
All rights reserved by the source institution and holding location. 

Resource Identifier: 
alephbibnum  001036266 oclc  18278439 notis  AFB8640 

Downloads 

Full Text 
OPTIMAL POLICIES FOR QUEUING SYSTEMS
WITH PERIODIC REVIEW
By
MICHAEL J. MAGAZINE
A DISSERTATION PRESENTED TO THE GRADUATE COUNCIL OF
THE UNIVERSITY OF FLORIDA
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE
DEGREE OF DOCTOR OF PHILOSOPHY
UNIVERSITY OF FLORIDA
1969
To
Roger Eric
.
ACKNOWLEDGMENTS
I would like to express my sincere gratitude to the
members of my supervisory committee and in particular its
Chairman, Professor Michael E. Thomas. His concern and
inspirations were invaluable toward the completion of this
dissertation. Also, I must thank Professor George Nemhauser,
first for original assistance in formulatin: the problem and
secondly, for his continual consulting.
One individual who has been foremost in never refusing
to offer any type of assistance is my office mate, John McKinney.
In spite of having to overcome the same problems as myself,
he always seemed to be there when needed.
I also would like to thank Lydia Gonzalez for so ably
typing my draft, and Mrs. Terrie Duff for her typing of my
final copy, overcoming difficulties of time and legibility.
Last, but most, I would like to thank my wife, Joan, who,
besides her forcitude and concern, has made im dedication
page possible.
TABLE OF CONTENTS
Page
ACKNOWLEDGIIENTS . . . . . . . . ... iii
ABSTRACT . . . . . . . . . . . .
Chapter
I. INTRODUCTION AND PROBLEM STATEMENT . ... 1
II. SINGLE SERVER MODEL. . . . . . . 5
III. COMPUTATIONS FOR SINGLE SERVER . . . 30
IV. MULTISERVER MODEL . . . . . .. 44
V. CONCLUSION ANID EXTENSIONS . . . .. 53
Appendices
I. PIECEWISE CONVEXITY . . . . . .. 56
II. EXTENSIONS OF THEOREMS TO MULTISERVER QUEUE 61
BIBLIOGPAPHY . . . . . . . . . . . 67
BICGRAPHICCAL SKETCH . . . . . . . .. 69
Abstract of Dissertation Presented to the Graduate Council
in Partial Fulfillment of the Requirements for the
Degree of Doctor of Philosophy
OPTIMAL POLICIES FOR QUEUING SYSTEMS
WITH PERIODIC REVIEW
By
Michael J. Magazine
August, 1969
Chairman: Michael E. Thomas
Major Department: Industrial and Systems Engineering
The general problem cf finding optimal policies for oper
ating queuing systems is investigated in order to determine
when to open or close servers so as to minimize cost. The
queue size is observed at equally spaced intervals cf time
and the structure of the optimal policy is determined.
The main restriction made, at first, is to limit the model
to a single server system. Justification for a form of an opti
mal policy is given. Then the proofs that this policy is in
fact optimal for the finite horizon, infinite horizon, and
averaging cost criterion are given. Also, conditions for
determining that the server should never be closed with custo
mers in queue are presented.
Two computational schemes for determining the optimal
policy are presented. The first, utilizing system parameters,
minimizes average cost by computing the expected waiting time.
The other solves the infinite horizon discounted cost criterion
by direct use of Howard's :!arkov programming algorithm.
The problem is then generalized to allow for a fixed num
ber of servers. Most of the results from the single server
follow immediately. The main difficulty is, therefore, to
establish and justify the desired rule so that it is analo
gous with the single server and is still as general as possible.
The primary mathematical concept used in solving the prob
lem is dynamic programming. This technique yields the solution
to the finite and infinite discounted cost criteria in both
models presented. Some of the results from replacement theory
help in solving the averaging case.
Finally, the conclusion shows how this work is an extension
of the related work in the field, and points out several exten
sions which naturally follow. The fact that the form of the
optimal policy is both intuitively appealing and easy to imple
ment makes the results useful in many queuing and related
systems, such as inventory and replacement models.
CHAPTER I
INTRODUCTION AND PROBLEM STATEMENT
Introduction
An important problem to consider is the economic behavior
of a stochastic service system. Most research in queuing
analysis has dealt with the underlying probabilistic structure
rather than identifying optimal policies for the operation
of the system. Within the operation of a service station
there is a fraction cf time in which the server is idle. This
idle fraction can be eliminated in one of two ways. Firs:,
the server can carry out auxiliary duties during this idle
fraction or, as an alternative, there can be a partial or
complete elimination of the station. We are concerned which
the latter, i.e., when to open or close our station in der
that some objective is optimized. The solution to this prob
lem is by no means immediately obvious,especially when there
is a statistical input of customers. In this paper we will
discuss a reasonable form for an optimal solution, prove
under what conditions this solution exists and discuss exten
sions including computational refinements for this type of
solution.
Previous Pesults
The justification for this work stems from the similarity
between queuing and inventory models. Much work has been
done using explicit cost functions on inventory models17
in particular getting optimal policies of a particular struc
ture, such as the sS policy. Because of this similarity in
structure it seemed reasonable to attempt to fit explicit
cost functions to the model with the hope of finding opti
mal policies similar in structure to the sS policy.
The mathematical background for this dissertation comes
first from dynamic programming2'13 and then from many papers
9,10
on Markovian Decision Processes.9,0 Many of these works are
11
discussed in a survey paper by the author. In a paper on
replacement policies which is an application of sequential
decision process, Derman5 defines a class of rules as con
trol limit rules. The form of this rule was helpful in ob
taining optimal rules for the queuing mcdel.
The earliest paper on optimization of queuing systems
6
was by Edie. He uses optimization techniques in determin
ing the best level of service based on traffic volume at toll
booths. Then the number of toll booths required at any time
of day can be established. Edie's results provided a minimum
number of toll collectors providing acceptable service with
relief for the toll collectors.
A paper by Yadin and Naorl7 was an important inspira
tion for this work. They dealt with the partial elimination
of a single service station under a (O,R) doctrine. This is
where the server when closed remains closed until R customers
are present in the system and remains open until the system
is empty. They give properties of this system under this
rule and also give computational aspects leading to a closed
form solution for the value of R.
Most recently Heyman and Sobell5 have recognized the
importance of explicit cost functions for stochastic service
systems. Heyman considers a single channel queue with Poisson
arrivals and arbitrary service time distribution. Upon com
pletion of arrival and service epochs he shows that for infi
nite horizon models there exists an optimal stationary policy
of particular form. He also finds an explicit cost function
and thereby is able to find an optimal policy as a function
of the parameters of the system. Heyman next considers a
twochannel model, one of which is always open, with the
added assumption that the service time distribution is expo
nential. He again shows what form the optimal policy must
take and uses Jewell's Markov Renewal Programming technique
to find the policy.
Sobel generalizes the problem by making the arrival dis
tribution arbitrary. Also, where Heyman uses a linear hold
ing cost, Sobel uses a very general holding cost. He then
shows existence for the same type of optimal policy as does
Heyman.
General Problem Statement
Consider "customers" arriving at a service facility from
an infinite population according to some statistically steady
input stream with distribution function A(t). Also, assume
there are at most s servers, each with distribution of ser
vice time Bk(t), k = 1, 2, ..., s. The system is observed
at discrete times (t, t2 ...). We are interested in first
showing the existence and then finding a policy of form (A)
so as to minimize some cost structure, where form (A) is:
(A) for each k (k = 1, 2, ... s) identify a unique
ordered pair [a*(t), b*(t)] such that if i, the number
in the system, falls between a*(t) and b*(t) a specific
action will be determined. If i does not fall between
these values then no action will be taken. By an action
we mean how many servers should be opened or closed .
The form of the optimal policy for each specific prob
lem will be of this structure and will also inherently
make it clear how many servers should be opened or closed.
This differs from the aforementioned works because of the
following refinements:
(1) Assume exponential interarrival and service distri
butions.
(2) Allow for a fixed and finite number of servers.
(3) Show the existence of an optimal policy for finite
horizon, infinite horizon and averaging cost criteria.
(4) Make the decision points at the end of time periods
of equal length. This is a major difference between
this work and the others. Observing at the end of
equal time intervals causes many difficulties that
are not encountered when observation points are the
completion of arrival or service epochs. These will
be pointed our later in the text.
CHAPTER II
SINGLE SERVER MODEL
Let us first consider a modification of this problem
which is far more tractable.
A Specific Problem
As a specific problem let us make the following changes
from the general problem formulation.
(I) Assume there is just one server. This is done
solely to simplify the problem,and hopefully will give insight
to the rrultiserver case.
(II) Assume the system has no bounds on capacity. But
also assume there is only a finite number of states for which
decisions will be made, i.e. when observed with more than M
customers, for some finite M, the decision will always be tc
open if we are closed. This is not a major restriction and
the reason for it will be pointed out later in the text.
(III) Assume stationary solutions. Blackwell assures
that both a* and b* will not be functions of time, in an
infinite horizon model, because of the finiteness of the
action space (here, only two possible decisions).
(IV) Assume both the times between arrivals and services
are distributed exponentially. Some comments are needed to
justify this assumption. A desirable characteristic of a
reallife service system is that the monitoring or checking
times are not too frequent. In fact, we would like to ob
serve our system at equal time intervals (once a minute, hour,
day, etc.). To ensure the problem is solvable the sequence
of states must form a Markov Chain (number of customers at
our service station must be independent of the past history
of the process); that is, there must be stationary transition
probabilities. This implies that the number of arrivals and
services must only depend on the length of the time period.
In order not to increase the dimensionality cf the state space,
A(t) and B(t) are restricted to be exponential distribution
functions. If, for example, B(t) has a general probability
distribution, to ensure the Mlarkovian property it would be
necessary to observe the system at the completion of each
service when it was open. We have chosen to be more general
in our observation points than in the arrival and service
distributions. However, much of the analysis and many of the
results can be used for the other, more general, case.
The specific problem can now be stated. Customers arrive
before a single server according to the Poisson probability
law. The time between services is exponential. The system
is observed at equal time intervals t = 1, 2, ... We are
interested in first showing the existence and then finding
a policy of form (A') so as to minimize some cost function;
where (A') is:
(A') identify a unique pair (j*, i*) with the interpre
tation that if and only if the server is open when
observed and there are at most j* customers present will
the system be closed and if and only if the server is
closed and there are at least i* customers present when
observed will the server be opened.
Note that this policy could be expressed precisely in the form
of (A) from Chapter I. If for k = 0, no servers open, we
choose (a*, b*) to be(i*, ) then the action will be to open
the server if the number in the system is between a* and b*,
and no action would be taken otherwise. Similarly, if k = 1
and (a*, b*) is chosen as (, j*) then the server will only
be closed when the number of customers is between a* and b*.
This differs with Yadin and Naor in that they assumed
a (O,R) doctrine and computationally found the optimal value
of R.
Characteristics
The properties of this model can now be given as follows:
(a) The Arrival Pattern  Customers arrive so that the
time between successive arrivals is distributed exponentially.
(b) The Service Mechanism  Customers are served in
order of their arrival so that the time between successive
services is distributed exponentially.
(c) Cost Structure 
(i) A Constant cost of opening a closed server
(ii) B Constant cost of closing an open server
(iii) C Cost of operating the server for one time
period
(iv) K(i) Holding cost, i.e., cost incurred when
i customers are observed in the system.
These costs may or may not be discounted with discount
factor a.
The decision process is as follows. The system is observed
at times t = 1, 2, ... and the state of the system Xt is noted.
Xt = i implies there are i customers in the system at time t.
The decision is then made to either leave the server in its
present state (opened or closed) or to open a closed server or
to close an open server, or if X > M and the server is closed it
must be opened.
The sequence of states {X.} forms a Markov Chain and changes
of states occur according to the following probabilistic struc
ture,
pij = Pr {X = jXt = i and the server is closed)
qi = Pr {Xt+1 = jIXt = i and the server is openI
Existence of a Control Limit Rule
Consider a single channel system with the aforementioned
characteristics. We would like to establish the existence of
a control limit rule. This is, if the server is closed show
that there exists a unique number i*, such that if the number
of customers observed in the system is no less than i* then
open the server. Further, if the number observed is less
than i* then keep the server closed. This rule has intuitive
appeal, saying, that if the server is not open with, say, 7
customers in queue then certainly it will not be open with,
say, 5.
It is desirable to show the existence of this number for
an averaging objective function. In doing so we obtain simi
lar results for both the discounted finite and infinite hori
zon models.
Dynamic Programming Formulation
Consider the solution to the finite horizon model using
dynamic programming. Define,
k (i,a) = minimum expected cost with n time periods
n
remaining when i customers are observed in
the queue using discount factor c, and k = 0,
1 depending on whether the server is closed
or open respectively.
Then the recursive equations are
o 0
o (i,a) = min K(i) + aZ p. 9 (j,a)
n j= ij n 1
K(i) + A + C + a q.j (j a)
j=0
where the alternatives are to keep the server closed or to open
it.
And 4I (i,a) =min K K(i) + B + a Z (a)
d n j=0
K(i) + C + C T a.. 1 )
j=0 
with alternatives closing the server or keeping it open, where
it is assumed that
0 1
0 (m,a) = K(C) + A + C + Z qmj mj,a) for m > M
j=0 mj
Define,
k
0(i,.) = K(i) k = 0, 1
i.e., the terminal cost of our process is the holding cost
for the number of customers observed. In addition, assume
(i) that K(i) is a nondecreasing function of i, a
reasonable assumption and one often made in inven
tory or replacement analysis.
and (ii) that for any nondecreasing function h(j), p..h(j)
j=0
and I qi.h(j) are nondecreasing functions of i.
j=0
This says, given some subset of states K = {k,
k+l, ...} the Chance of entering K is nondecreasing
in i.
Consider the server closed. For n = 1 we have
S(i, a) = min K(i) +a Z pi K(j)
j=0
(1)
K(i) + A + C +ca q K(j)
j=0 1j
From assumptions (i) and (ii) we know both of these alternatives
0
and also, (i,a) are nondecreasing. If at i = 0, the second
alternative is chosen then we must ensure it must be chosen
for all i (always open server). If at i = 0 the first alterna
tive is chosen then these t,'o nondecreasing curves may cross in
at most one place, the i* being sought. If they don't cross
at all before M. then the server will never be opened until
at least M customers are present. The possibilities are
illustrated in Figure 1. To ensure case c cannot occur an
additional assumption must be made.
COST
a i
COST
a Never open server
c Undesirable case
  cost for closed server
cost for open server
p
p
p
p
p
d i* i
b Never close server
d Unique opening point
Figure 1: Associated costs for two alternatives
p
COST
COST
We should comment here how Heyman's model, that of check
ing after each arrival,fails to encounter this difficulty.
Let us say that case c denotes the graphs of the curves in
Equation 1. Let i be the first crossing point and i2(>i1)
the second, that is keep the server closed for i
i >i2 and open otherwise. If the system is observed at i
and then at each arrival thereafter, as Heyman does, as soon
as il is reached the server will be opened and the state i > i2
for a closed server will never be reached. Checking at dis
crete time points, however, may result for some t in
Xt = i for i < i1
and Xt+1 = j for j > i2
resulting in keeping the server closed at both points, and
hence his rule of always opening after iI may not hold.
Additional Restriction
As so often happens in this type of problem when a barrier
is reached, an additional condition must be imposed. By anal
ogy to the definition of convexity we impose the following
condition:
8K(j) + (18) K(i) > K.(j + (le)i) for
8e[0,l] and the argument for K(.) integral (2)
For obvious reasons we call (2) the convex property. It would
be possible to achieve the same results by defining the envelope
of the function as convex. However, there is very little addi
tional computation in analyzing the discrete function.
Uniqueness Proof for n = 1
Recalling that
0
S(i,a) = min K(i) + a Z .K(j)
'1 j=0o
K(i) + A + C + e F q..K(j)
j=0
it must be determined when
K(i) + a Z p. K(j) () K(i) + A + C + a q jK(j)
j=0 j =0
or
SC + A
Z K(j) [pij q ij] A
j=0 _
This will be done by showing that
Z K(j) [ij qij]
j=0
is a nondecreasing function of i and hence can cross the con
stant line A at most once.
Let x and y be the number cf arrivals and services,
respectively, in a given time period.
Then
Z p K(j) = Z Pr (number of arrivals equals x) "K(ix)
j=0 x
= E K(i + x) = E E K(i + x)
x x y
and also,
Z qi. K(j) = I: Pr(x arrivals) Pr (y services)
j=0 x y
K(i+xy)
= E E K(i+xy)
x y
where
K(n) = 0 for n < 0.
Since K is nondecreasing
D. E E [K(i+x) K(i+xy)] > 0[1]
or,
or,
Z K(j) [Pij
j=0
 q i >_ 0
ij 
Now, allowing j = i + h, (2) can be rewritten as
6K(i + h) + (1 e) K(i) > K(ei + eh + i ei)
= K(i + oh)
6K(i + h) gK(i) > K(i + ah) K(i)
To say this we must ensure that both 7 Pi K(j) and
7 q .K(j) are finite. To show convergence employ the ratio
test as follows:
SPijK(j) =
j=0
S (Ji) e\
ji (j K(j)
j=1
so that
Pji+1 ;\
,i+ (ji+1)!
ji+1 .\
Pi, K(j) = (i+l
1,i(ji) a
k(j+l) and
nd K(j) and the limit as j of
piz11K(j+l) N K(j+l) must be <1. This holds for
P K(j) (Ti+1) K(j)
1i,j
a wide class of functions such as exponential and polynomials
and will be assumed true. Certainly, then, qij.K(j) will also
converge as it is less than or equal to 7 pijK(j).
or as an alternate definition
K(i + h) K(i) > [K(i + h) K(i)] (2')
which must hold for all 0 < <_ 1 and eh integer. In particular
it must hold for Oh = 1, i.e.,
K(i + h) K(i) h[K(i + 1) K(i)]
for all integral h. As a special case when h = 2
K(i + 2) K(i) > 2 [K(i + 1) K(i)]
or
K(i + 2) 2 K(i + 1) + K(i) > 0 (3)
Denote by (j)K as the jth difference of K(i). Then
(1)
A K = K(i + 1) K(i)
S(2)K = (A(1)K) = K(i + 2) 2 K(i + 1) + K(i)
Notice, however, that this second equation is just equation (3).
Hence, we have proven the following lemrra.
Lemma 1: The second difference of functions with the convex
property is nonnegative.
Therefore, we can conclude that
A (K = K(i + 1) K(i) is nondecreasing. (4)
Now consider
(1)
A ( [K(i + x) K(i + x y)] =
K(i + x + 1) K(i + x) K(i + x y + 1) + K(i + x y)
But this is nonnegative since
K(i + x + 1) K(i + x) K(i + x + 1 y) K(i + x y)
from the property in (4). Hence,
K(i + x) K(i + x y)
is nondecreasing in i for all x and y. Then surely,
D. = E E [K(i + x) K(i + x )
i x y
must be nondecreasing in i and hence crosses A at most once.
This proves the following important theorem.
Theorem 1: For r.=l,10 (i, a) has a solution given by a control
limit rule, i.e., there exists a unique number such that
if and only if with one period remaining there are more
than this number in the system, will a closed server be
opened.
For the case when the server is open with one period re
maining,
K1(i, a ) = min K(i) + B + i Pij K(j)
j=0
K(i) + + Cat q K(j)
jij IJ i
j=0
This leads to observing when
K(i) + B + a I pij K(j) () K(i) + C + a qij K(j)
or,
S [pij qij] K(j) () C _=
j=0
The preceding theorem then also applies here, hence guaran
teeing a unique point at which the server would be closed.
Proof for arbitrary n
For n>l it must be shown that (j*, i*) policies are opti
mal for,
0 0
n (i, a) min K(i) + a p (j, a)
n j= j n
1
K(i) + A + C + a qij n (j, )
j=0
and
1 0
4 (i, a) = min K(i) + B + a 4 p n (j, a)
n ij n1
1
K(i) + C + a Z Pij (j, a)
Before this is shown, let us prove the following lemma.
Lemma 2: For n=l, i* > j*
Proof: For n=l, i* must satisfy
K(i') + c Y Pi*j K(j) K(i*) + A + C + a E qij K(j) or,
A+C
i*j qi*j] K(j) >+C
Similarly j* must satisfy, K(j*) + B + ar pj.j K(j)
< K(j*) + C + Z qjj K(j) or,
S[p qj*j] K(j) CB
S
Since A + C > C B and Z [pij qij] K(j) is nondecreas
ing, i* j* must hold.
Now let us establish the existence of the values (j*, i*)
for n=2. We have that
0 o
02(i, a) = min K(i) + a pi 40 (j, a)
(( 5)
K(i) + A + C + a Z qij l (j, a)
and
1 0
4 (i, ) = min K(i) + a pij (ja) + B
(6)
K(i) + C + a z qi (j a)
Just as before we can write
0 0
Pij 4 (j,a) = E E (i+x,a)
ij 1 x y I
and
1 1
1 qij l (ja) = E E Vl(i+xy,a),
0 1
and,as before, if it can be shown that 4l(i+x,a) l(i+xya)
is nondecreasing, then so is
0 1
S[Pij ( a) q (j', )]
which implies that the alternatives in (5) and (6) can cross
at most once and the result will be established.
.l(i+x,a) = min K(i+x) + i a i+xj K(j)
K(i+x) + A + C + aZ qi+x,j K(j) (II)
min K(i+xy) + B + a Z pi+x,j K(j) (III)
S= (i+xy,a)
K(i+xy) + C + i+xy,j K(j) (IV)
where the (I), (II), (III) and (IV) notationally correspond
to that particular alternative.
It is necessary to consider only the possibilities when
the minima are (I) and (III), (I) and (IV), and (II) and (IV),
i.e. (II) and (III) cannot concurrently be minima by lemma 2.
Now, let us formally prove the following.
0 1
Theorem 2: (i+x,a) (i+xy,a) is nondecreasing.
Proof: When (I) and (III) are minima 1 (ix,a) (i+x )=
K(i+x) K(i+xy) + a Z[i+x,j i+xy,j] K(j) B
which is nondecreasing, the first two terms from assump
tion(i)and the second two since K(i+l) K(i) is nonde
creasing.
When (I) and (IV) are minima,
0 1
1(i+xa) (i+xy,a) = K(i+x) K(i+xy)
+ a1 p q) C
+ aZ [Pi+xj i+xy,j K(j) C
which is nondecreasing. This can be shown be replacing
the bracket by
[Pi+xj Pi+xy,j + Pi+xy,j qi+xy,j] K(j)
which is the sum of two nondecreasing functions.
Finally when (II) and (IV) are minima
0 1
4' (i+x) (i+xy,j) = K(i+x) K(i+xy)
+ a qi+x,j i+xy,j] K(j) + A
which is nondecreasing as was the first case considered.
This completes the proof.
This, then,leads to the following:
Corollary 1: There exist numbers (j*, i*) for n=2 such that
(i,a) and 12(i,a) have optimal solutions of the required
form.
Now, let us introduce the hypothesis that,
0 1
for n=3, 4, ... k,4 (i,a) and (i,a) have control limit
n n
rules and, also,
S 0 (j,a) (j qa)]
Z [Pij 1ni 1ij nl '
is nondecreasing.
It becomes necessary to show next that
0 0
+(i,a) = min K(i) + a ~ pij k(j,)
K(i) + a qj q(ja) + A + C
and
1 0
k+l(i,a) = min K(i) + B + u p ij (j,a)
k+1 k
K(i) + C + a qj .(ja)
have the required property. This is done in precisely the
same manner as before, i.e., it is shown that
0 1
i [Pij k(jr, ) ij k '( )
is nondecreasing by looking at
0 1
0k(i+x,a) k(i+xy,a)
k k I
Again, we shall first show
Lemma 3: For each n, i* > j*.
Proof: At i*, the following must be true,
0 1
K(i*) + a i jnl (j,) > K(i*) + A + C + qijnl (j)
or
0 1 A+C
S i(ja) q i n (j'a)] 
i [i j n i j n1 
At j* the following must hold,
0 (j) K(j*) + C +
K(j*) + B + apXp j' (j'a) jjK(j*) + C + pj j
Jni 1 CB
jij n jj l(j,) a
Since A + C > C B and the left sides are nondecreasing
by the induction assumption, it naturally follows that
i* > j*.
Theorem 3:
k0(i+x,) = min K(i+x) + a Pi+x,j k(j,a) (I)
SK(i+x)+ Lx 1
K(i+x) + a q k(ja) +A + C (II)
i+x,j k1)
min K(i+xy) + B + a i+xy ) (III)
K(i+xy) + C + ai q., ( 1(j,a) (IV)
i+xy,j kl (IV)
(where the (I), (II), (III) and (IV) are unambiguous) is
nondecreasing.
Proof: From lemma 3 we see once again that it is only neces
sary to consider when (I) and (III), (I) and (IV), and
(II) and (IV) are minima.
When (I) and (III) are minima the difference is
K(i+x) K(i+xy) + al [pi j Pi] ) B,
i+x,j i+xy,j k1
which is nondecreasing; the summation is nondecreasing
which follows from the discussion on piecewise convex
functions in Appendix I.
When the minima are (I) and (IV) we have
K(i+x) K(i+xy)
0 1
Sa [Pi+x,j 9k1(j'a) i+xy,j kl(j'a)] C.
This is nondecreasing since
i+x,j l(j,a) qi+xy,j k. (j'a)
Si+x,j Pi+xy,j] (kj'c)
0 1
i+xy,j k 1') qi+xy,j 1 (j,a)
and the first two terms are nondecreasing by the first
part of this proof and the second two by the induction
hypothesis.
Finally, when (II) and (IV) are minima
K(i+x) K(i+xy) + a [qi+x jqi+xy,j ]k (jj, ) + A
is nondecreasing by again using the results of the dis
cussion from piecewise convex functions. This completes
the proof.
0 1
Hence, it is established that +l (j,') and .k+l(j c)
have control limit optimal rules.
Thus, we have established,
Corollary 2: For any n, t 0(i,c) and L (i,a) have solutions
n n
which are control limit rules.
Extension to Other Objectives
The theory behind the next result is due largely to Bellman.2
Theorem 3: A control limit rule is optimal for the infinite
horizon model.
0
Proof: Let us prove this theorem for 4. (i,a); the other follows
similarly.
I
We know that
0
n (i,a) = mi
n
SK(i)
K(i)
Define,
Tl 4 ) = K(i)
1 n1
+ ij n1 ( )
1
+ A + C + E qi.. nl(j,')
:1j n1
0
+ Z Ppij n (j ,o)
l]n1
and
0
T (9n_ )
2 n1
= K(i) + A + C + a Z qj (ia)
n1
Then,
O (i,a) = min
n j=1,2
[T (1 1
j n1
Let j(n) = index which gives the min [T. (4n)]
j=1,2
which implies that
0 0( )
n j (n1) n1
j
STj(n) ('n
and
o n
4n (ia) = Tj(n) ( 0)
n+1 j (n) n.
S (nl)
 j(n1) n
Subtracting,
n + (ni) n 'n n
ln "I 0. 0 )n T (i^)
0 0
Tj(n) ( ) Tj(n) (n )
0 0
< max[IT( 0 nT ( ]
j=1,2 nj
0
< max[ K(i)+a Pijcn l(j'a)
) )n1
K pi jp 0(j,a)I ,K(i) +
i n
1 1
A+C+a Z qij (j, a)K(i)ACa Z1 q (j ) ,
< max[la i p n (j,)a Z p n(j,) ,
< p  (ja) (j,a)
ij nl n
Hence
max (i,0a
max n n+1 (i) 1 Pij n n '
i i
0 0
n1 n
Define
u = max[ (i,) (ia) ,]
n n n
1
and hence
2 n
u
n+1 n n1 1
and therefore
GO 00 u
S n 1
E u < Z a u 
U n+l i 1 = a
n=0 n=O
and we conclude that
0 0
E n (ia) (i, )]
n+1 n
n=O
converges uniformly and that 0 (i,a) converges to t (i,a)
n
for all i, where 0(i,a) satisfies,
0 (i,a) min (i) + aZ p j (j,a)
K(i) + A + C + a Z qij (j,a)
completing the proof.
Now the significance of restricting our decision set to
be finite when the system is closed will become apparent. We
shall assume that for any c>0, an M(c)3: Pr[X > M(t)]
and,henceforth, consider opening only in a finite number of
states. When the server is open, assuming that the arrival
rate is less than the service rate is sufficient to guarantee
that the queue size will remain finite; hence, only a finite
number of decision points can occur.
Finally, we can prove the following:
Theorem 4: A control limit rule is optimal for the averaging
cost criterion.
Proof: Let i be the optimal crossing point of 0 (i,a). Let
{as}, with lim as = 1, be a sequence such that i = i*
S+= S
for all s. Since there are at most a finite nunmer of
states where the system will be opened, such a sequence
and i* must exist. Let R be any rule which is not con
trol limit. Then certainly
0 > 0 U
Hence,
0 0
lim (las) R (i,as)> lim (1as) (i,as
ss sR s
and from Arrow, Karlin and Scarf,
S= lim (1i ) 0 (i,a ) where
R s Ri s
S
0 1
R = lim Z g(Xt) and where
Tm t=l
g(Xt ) denotes the cost incurred when observed in state
X Therefore, %. where is the optimal value
STherefore, R 
under the control rule i* and hence i* is the optimal
1
rule. The proof is identical for .
Closing When There is a Nonempty Queue
Many have conjectured that a server will only be closed
if no customers are awaiting service. Others have said that
the closing point is some function of the system parameters.
Let us answer this question by showing under what conditions
the following theorem is true.
Theorem 5: The server will never be closed when any customers
are present in the system.
Let us assume that we have established a unique point,
say i*, at which the server should be turned on or opened.
Let us also assume, contrary to the theorem, that the optimal
operating policy,n, designates to close the server at some
number j*, 0 < j* < i*.
Since the process is in equilibrium, we know that with
probability one, state j* (or less) will be reached once the
server has been turned on. Hence, the process will alternate
between states j* and i*.
There exists, however, a number k, 0 < k < j*, such that
the expected cost under policy n is equivalent to the expected
cost under policy ni, where,
nI: Open the server when there are i* j* customers in
[2]
In fact it will alternate between states less than or
equal to j* and greater than or equal to i*.
line, close when the line is empty and keep k3 customers
on the side at all times.
Define now policy 12 as follows.
H2: Open the server when there are i* j* customers
in line, close when line is empty, and after some finite
time serve the k customers.
Now for each possible objective function consider the
following three policies.
1) Total Expected Undiscounted CostHere the additional
expected cost under nl, that of holding the k customers to
the side becomes infinite, while the additional expected cost,
that of serving the k customers, is certainly finite and hence
12 dominates H1 and, therefore, also R.
2) Average CostThe holding of the k customers under
policy H1 occurs for each period and hence adds a positive
quantity to the average period cost. However, the cost of
serving the k customers occurs for only a finite number of
periods and hence adds nothing to the average period cost.
Hence, again n2 dominates both H and Hn.
3) Total Expected Discounted CostHere, both the holding
cost of the k customers along with the cost of serving these
k customers is finite and hence a comparison is necessary.
For a discount factor a the cost of holding the k customers is
at least n K
n K(k)
K(k) E a 
1E
n=0
3 If k is not integer keep instead [k] customers off to the
side, so if anything H1 dominates H.
The cost of serving the k customers, which we assume takes
i periods, along with the holding ccst of these customersis
bounded above by
i1 11
K(k) E aU +C E rn
n=0 n=0
= [K(k) + C]
1 a
To choose 2 over n we must require that
Kk)1 a K(k)
[K(k) + C]
1 a 1 a
2 C
& K(k)+C
Hence, if the discount factor is large (a reasonable assump
tion), 2 dominates H as well as nR.
4) Finite HorizonIn this case it is possible to close
the server with customers present. This event occurs if the
holding cost is small compared to the operating cost.
Again, the difference between analyzing my system and
that of analyzing the system after each arrival and departure
is that k in the preceding argument is just j*.
Concluding Remarks
Finally we can say for the single server queue that an
optimal stationary policy always has one of the following forms:
a) Always keep the server closed until M is reached.
This case arises when discounting is used and the holding cost
is small compared to the operating cost.
29
b) Never close the server. This may occur when the
server setup and shutdown costs are large compared to the
operating cost.
c) There exists a pair (j*, i*) with the interpretation,
open the server only if there are at least i* customers present;
close the server when there are no more than j* customers
present. The important special case of j* = 0 is discussed
in Theorem 5 and that which followed.
CHAPTER III
COMPUTATIONS FOR SINGLE SERVER
Once existence of optimal solutions has been established
it becomes necessary to find an i* which might make it worth
while to "dismantle" the service station. The closing of the
service station under a (0, i*) doctrine will cause increased
holding, and setup and shutdown costs. The savings from
the partial elimination of the station, during an idle period,
might not outweigh these increased costs. Hence, an optimal
i* must be found and the cost therein achieved must be com
pared to that of a simple queue, i.e., never close the server.
A simple computational scheme is developed to make this com
parison.
The second part of this chapter makes use of Howard's
approach in analyzing the system from a computational view
point. Description of the algorithm, calculation of probabil
ities,along with some interesting results are presented.
Averaging Computational Scheme
Calculation of Average Queue Length
Let us consider an averaging cost criterion, i.e., find
ing the optimal policy for the average cost per cycle. By a
cycle we mean the following:
Let us begin at the moment our service station has been
closed dc::n there are no customers waiting for service.
Call phase 1 that time it takes for i customers to accumulate
before the service station. By the second phase, is meant
the time from when the ith arrival enters the system until
the end of the present time period. Let the average number
of arrivals in this phase be denoted by a'. Let the third
phase consist of servicing these i + a'customers along with
their generation arrivals (all arrivals occurring while the
original i + A'are being served) until there is no queue and
our service station is once again closed down. Phases I, II
and III make up one cycle.
The average time for phase I is i / k, the assembly time
for i customers.
Let F(t) be the distribution of the length of time for
phase II. Let E(t) be the expected length of time for phase II.
Then,
\ i t (t)j
ja= je' e dF(t) = t dF(t) = \E(t)
0 j=0 J'
and hence E(t) =
Let,
x (1)
be the traffic intensity. In addition define P, and P as
D C
the time of bus and clear periods for ordinary queuing sys
tems. Then, because of the Poisson arrival process,
P = 1/\
c
Also, the fraction of time the server is busy is given by
Pb
P +P
b c (3)
On combining (1), (2) and (3), we obtain,
1
P 1
b P,
This is the expected length of the busy period for one cus
tomer and, hence, the expected duration of the third phase
is (i+&')/(u.).
The average length of a cycle can then be easily obtained
by summing,
i (i+ ') i (P.)+a' (pA)+(i+C')A
T + +
A iaA A (v\)
u (i+L') i + C'
.\ (1.\) X (lp ) (4)
Also, the fraction of time the system is in each of the phases
is,
i i (10)
AT i + cJ (5)
CL C'(1c)
AT i + C' (6)
i + (7)
(pA)T 
for phases I, II, and III respectively.
Let us adopt the following notation:
Pkj = Pr There are k customers in the system and there
are j available servers
K = 0, 1, ... and j = 0, 1
ttA random variable denoting the length of time from the i
arrival to the end of the next observation point.
T A random variable denoting the time between the
arrival of a customer in the third phase until the com
pletion of the particular service in progress.
T A random variable denoting the time between the
arrival of a customer in the second phase until the
end of the second phase.
W Average queuing time
L Average queue length
Let us note some important relationships:
Since phase I is just the arrival of customers through
a Poisson stream all states in phase I will occur with the
same probability. Hence, from (5)
P 0 K = 0, 1, ..., ii (8)
KO i+ '
Also, from (6) the probability of being in phase II is
= _(lp
.PKO J +. '
K=i (9)
Finally, the associated probability for phase III is,
SpK1 = (10)
K=l
Next, Cox and Smith4 show that the forward delay, T, is
distributed so that
2
E(T) = E(t)(
2 (11)
where t is the original random variable and v, the coeffi
cient of variation of t. For the exponential distribution,
v = 1.
Now, we proceed to analyze the queuing time of customers
arriving in different phases. For a customer who arrives in
phase I, where there are K customers present, his waiting
time is,
(ilK)/\+ E(tt) + (K+1)/p
.An arrival in the second phase when the system is in state K
can expect to :wait,
E(t ) + (K+l)/w
Finally, a customer in the third phase, when K are present
at his arrival, will wait on the average,
E(T ) + K/u
Now we are in the position to write down the average
queuing time W:
ii
W = Z pK [(i1K)/\ + E(tt) + (K+1)/p)]
K=0
+ L pKO [E(Tt) + (K+1)/u]
K=i
+ Z Pj [E(T ) + K/p]
K=l J s
Insertion of (1), (8), (9), (10), and (11) yields,
= 1 i(i1) (1c')
19
S== [(L+1) +
A 2(i+a')
'i( 1 ) ~2a (1P) ( 1+.
+ +
i+a' i+a' 2
From Little's formula,
L = .W
the average queue length is then,
2 i+v
P i(il) ai
L = p + + + [i+ar ()]
10 2(i+a) i+a' 2
which is the average queue length of a simple queue, given
by the first two terms (PollczekKhintchine), plus the addi
tional queue length due to the intermittent operation of the
server.
Optimization
Now it is possible to write down the cost function for
the averaging criterion.
First, in one cycle a cost will be incurred due to
T
setup and shutdown of the server. Also we can expect a con
tribution of K(L) as the average holding cost per cycle.
Finally, in one cycle there is savings due to the decreased
use of the server. The server is being used only when in
phase III and, hence, the cost incurred from its use is expected
to be oC/T.
Therefore, the average cycle cost is
Ct(i) = K(L) + (l A+B), C/T
To find the optimal opening point, i, we may differenti
ate Ct with respect to i, set equal to zero, and solve for i
(an integral value is naturally required). However, this is
impossible as given, as it is necessary for the form of K()
to be explicitly given.
A Specific Case
First, for the sake of example cnly, and second, to com
pare the results with some previously cited works, let us
assume that the holding costs, K(), are linear with respect
to the averaging values. Then,
2 lt1+v
D i (il) O' t
(i) = K [o+ + + [i+a'( )]]
t i) 2 (i+a') 2
+ (A+B)(1lp)
+ PC/T
Differentiating,
1 Ki
SK. (i+.') (i1) (i1)
C 2 2
ai (i+ ') 2
2
l+v
+ K.(i++') L[Kai + Ka'2 t (A+)
Ki2 + 2K + K' 2 (A+B) (1,A+) 0
2t
(i+, 2 (i+d)
or,
Ki + 21KP + K,'['1v2) 2(A+B) (lp) = 0
or,
t K
which has roots,
( 2t' 4a2 4 [ (,_1 t2) 2 (A+B), (1 )] )/2
t K
or,
S \/,,2 [,( , vt2 2 (A+B) A (1C )
St K
Simplifying we obtain,
_2.
2(lP) (A+B) + 2'(1
K t
The positive root yields the minimum and, hence
i = '+ 2(1)(+B a(l+a2) (12)
The optimal solution is then obtained by comparing the
cost of the nearest positive integers [i], [i+1] and the pos
sibility of permanent server availability. This latter cost
is just
2
C = K [p+ ] + C/T
s 1p
That is, i* is defined by
C(i*) = min [C[i], C([i+l]), C ]
(in the last case i* = 0).
It is interesting to note that the value of i obtained
by differentiation is not a function of the running cost, C,
of the server. Actually, this result is not surprising as
over a reasonable length of time the same number of customers
will be served on the average; hence, the server will be running
for the same length of time.
Finally, if the system is observed after each departure,
rather than at equal intervals, phase II disappears. Therefore,
(12) becomes
S(A+B) \ (1c)
V K
This result agrees precisely with that of Heyman's.
Computational Experience Using Ecward's Algorithm
Brief Description of Algorithm
Given a process which recieves a reward qi at the begin
ning of a period when found in state i, then, if Vi (n) is
the present value of the total expected return when in state
i with n transitions remaining, we obtain
M
V.(n) = q. + Z [pij V (nl)]
j=1
for each i and n where a is some discount factor and a total
of M states.
9
With the introduction of alternatives, Howard has devel
oped an algorithm which minimizes these present values for
an infinite horizon problem, i.e., replacing Vi = lim V (n)
n*=
for V.(n). This routine can be summarized by the following
flowchart in Figure 2.
The iteration process can be entered in either box and
Howard guarantees that each cycle will yield a present value
at least as good as the one previously obtained. He also
guarantees that there is convergence to an optimal policy
and this is noted by having policies on two successive itera
tions identical.
In the problem at hand either of two decisions may be
made in any of the 2:1 states. The states are characterized
[1] Veinott 16 has guaranteed convergence in a finite number of
steps.
Figure 2: Policy Interation with Discounting
by the number in the system 0, 1, ..., M1 and whether the
server is closed or opened. The cycle was entered in the
policy improvement routine by choosing all the present values
as zero. The first step, therefore, was minimizing immediate
returns, i.e., choose the alternative that chooses
K(i)
q. = min K(i)
K(i) + A + C
when the server is closed and
K(i) + B
q. = min
K(i) + C
when the server is opened. The new present values were then
calculated and iteration process continued until convergence
was obtained.
Calculation of Probabilities
In order to make use of the algorithm explicit values
for the transition probabilities,pij and qij had to be known.
When the server was closed, p.., the probability of going
from i to j in one time period,was simply the probability of
ji arrivals, where the arrivals are Poisson distributed with
parameter .\
With the server opened, the probabilities qi are derived
14
as follows. Prabhu finds the Pr (there are
at time t I there are i in the system at time 0)
= Pr(Q(t) 1 j Q(0) = i)
e(e+u)t ab ta+b e(\+p1)t a+j+l bjl ta+b
A ji) aab! (ji2) a!b'
where A(j) denotes the set of integers a, b such that a>0,
b>0 and ab
Subtracting from this quantity
Pr (Q(t) < j1 I Q(0) = i)
we get
Pr (Q(t) = j I Q(0) = i)
Next, setting t=l, i.e., one time period, we establish
q eha b a b a+j+l bj1
qi = e (+ LL + Z i }
ij a!b! abi) a a!b!
1 ab=ji i1) a A(ji2)
Since we always make the decision to open at state *1 and
above, these probabilities are truncated at M!. This is neces
sary as Howard's routine allows for only a finite number cf
states.
Results
A FORTRAN program was used to calculate these probLailities
and then incorporated into another which was Howard's policy
improvement routine. Some of the results obtained were both
reassuring and enlightening. Since, however, sensitivity on
one parameter was accomplished while the other parameters were
fixed at particular levels any definite conclusions were dif
ficult to reach.
Sensitivity to discount factor
Theorem 5 in Chapter II stated that it was possible to
close the server with customers present for the infinite
42
TABLE 1
SENSITIVITY OF DISCOUNT
FACTOR TO HOLDING COST
FOR i=1.0, u=2.0,
A=10, B=10 AND C=40
j* for K(X) =
a : X2 + X e"
0.9 0 0 0
0.8 0 0 0
0.7 M 0 0
0.6 M 0 0
0.5 M 2 0
0.4 M 4 1
0.3 M M 2
0.2 M M 3
0.1 M M 4
horizon discounted case. For fixed costs, however, this was
only true for small a. Table 1 gives j*, the closing point
as a function of three different parameter values. Notice
that the values are not only small but also as the holding
costs became "more convex" the value of the discount factor
when the server is closed with customers present decreases.
Other results
Certainly extreme cases led to extreme results as far
as determination of i* and j*. For example, for high setup
and shutdown costs and low operating costs as soon as we open
we always remain open. Also, if the holding cost is small
compared to the fixed costs once we close, we will always
remain closed (since the fixed costs are constants, this only
occurred in our investigation when the holding cost was linear).
An interesting observation was that the sensitivity of i*
to the ratio of the arrival and service rates (A/,) was very
slight. However, i* was quite sensitive to the magnitude of
X and p. When A and p were chosen large (>1) i* turned out
to be only 0 or 1. However, when their magnitudes were decreased
(A=0.1, p=0.2) i* had the value 4. Certainly there is varia
tion here depending on costs but large arrival and service
rates imply larger shifts in state. This means having large
holding costs and soon after no holding costs and conversely.
This implies many setup and shutdowns and hence i* is very
low so that the server would only be shut down when empty.
CHAPTER IV
MULTISERVER MODEL
Introduction
Let us return now to the more general problem, that of
finding optimal policies for the operation of a service mecha
nism with an arbitrary number of servers.
The assumptions made at the beginning of Chapter II will
still hold, except the following change:
(I') Assume there are s servers. Their service times
are independent exponential random variables each with
rate p. Customers are still processed in the order of
their arrival. If a server is closed down while proc
essing a customer, that customer rejoins the queue.
Also, just as in the single server case, we shall assume
that for k, the number of open servers, our arrival rate is
such, that given any E>0, there exists and M(E) such that:
P[Xt>M(E) ]<
This along with the assumption that s.\
decisions will only have to be made in a finite number of states.
The cost structure is affected only in that there is now
an operating cost, setup cost and shutdown cost for each
server, respectively C, A, and B. The transition probabilities
will now be denoted as
k
i = PrXt+l = j = i and k(=0,l,..., s)servers are opened}
where Xt is again the number of customers in the system at
time t.
Form of the Optimal Policy
In this section we will describe a policy which we will
define to be optimal. This policy most closely resembles that
of the (j*, i*) for the single server.
Given the state of the system,we are interested in deter
mining unique numbers which tell us how many servers will be
open after a decision is made. Notationally, let us describe
an optimal policy as follows:
*
(i,k : j J ,..., k1; ik+l '' is) (1)
where the i, k represents the present state of the system,
i.e., i customers present and k servers opened. The sequences
*
(j } and {i} are nondecreasing sequences with the following
9
interpretation:
*
If i
m1 m 
*
after decision). If j _lim, close km servers
*
(have m open after the decision). If i>i or i
all the servers open or closed respectively. If i does
not meet any of these conditions keep k servers open.1
[ Note how this conforms to the original form for an optimal
policy described in Chapter I.
Figure 3: Costs associated with a multiserver queue.
An optimal policy may permit going from j to j+l or j to
j1 for all j. If this is the case both sequences are increas
ing. However, an optimal policy may dictate a state where
the decision to open only one server will never be made. In
*
this case i+ = iz (or, possibly j = jj. This is best
illustrated by a simple example.
Let us assume that s=3 and presently i=0. Figure 3 depicts
two possibilities as to what the cost might be as a function
of the number of servers to be opened.
*
In (a), il, i2 and i3 are well defined to be the crossing
points of the (0,1), (1,2) and (2,3) intersections respectively;
and the sequence of i's is increasing. In (b) it is never opti
mal to have one server open so in order to conform to our rules
*
our optimal policy will have i = i2 and the sequence of i's
is nondecreasing.
Dynamic Programming Formulation
Now that the elements of the model are given let us again
set up the dynamic programming formulation for the finite hori
zon.
Define n(i,a,k) as the minimum expected cost with i
customers present, k servers opened with n periods remaining
in our horizon,using discount factor a.
Then our recursive relationships are:
K(i)+kB +a p (j,a,0)
ii'nI
1
K(i)+(k1)B+C+a E P jn0(j,,1
ij ni
k
(i,a,k) = min K(i)+kC +a E p i4 n (j, ,k) (2)
K(i)+ (k+l)C+A+a 1 pij .+I (j ,a,k+l)
ij n1
K(i)+sC+(sk)A+a ,p p (jcs)
ij nl
where
0 (i,a,k) = K(i)
and as an analogous assumption from the single channel system
we can say that there exist numbers M, M, ... Ms which
demand that
k+l
(m,a,k)=K(m)+(k+l)C+A+a L p (ja,k+l) for mLMk (3)
n m] nj1 k
where there is a onetoone correspondence between the k and
Mk Since this sequence (Mk} is finite there is certainly a
maximum and for simplicity we will impose (3) for m > max {MIk}=!.
All this says that whenever M customers are reached and k ser
vers are opened we will automatically open one more server.
Furthermore, the assumptions that K(i) is nondecreasing
and convex will again be made, along with assuming that for
each nondecreasing function h(j)
SP.k h(j)
j=0
is nondecreasing in i for each k.
Proof for Finite Horizon
The same sequence of developing is used for the multi
server system as was used for single server. In this chapter
only this logical sequence of steps will be mentioned to ensure
our results for an arbitrary number of servers. Proofs for
the theorems will appear in Appendix 2.
For n=l, let us illustrate how the result carries by
considering s=2. Then,
1 (i,c,0) = min
#l(i,c,l) = min
l,(i,a,2) = min
when the server
0
E K(j)[p
ij
and that
K(i) +C K(j)
SK(i)+A+C +L. E p1 K(j)
2
K(i)+2(A+C)+cE p i K(j)
ij
K(i) + B C+ E p K(j)
SK(i)+C+a 2 pi K(j)
2
K(i)+A+2C+a p 2 K(j)
0
K(i) + 2B + a pi K(j)
1
K(i)+B+C+a C pj K(j)
2
K(i)+2C +a p K(j)
is closed it must be shown that
 p .] is nondecreasing
ij
SK(j)[ 2
(j). Pij] is nondecreasing
13] 1j
(The sum of these two is just
0 2
E K(j)[p p. ]
ij 13j
which would certainly be nondecreasing as it is the sum of
two nondecreasing functions.) Doing this leads to proving
Theorem 1: For n=l the policy of form (1) is optimal.
To extend the results for arbitrary n, let us point out
the steps that make the analysis for the single server and
that for an arbitrary number of servers similar.
For n=2 it becomes necessary to show that
kl k2
U1(i+x F y.i,,kl) 1(i+x E yi ,,k2
i=0 i=0
for each kl and k2,such that k2 >kl, is a nondecreasing func
tion of i. This is equivalent to saying that there is at most
one crossing point for any two alternatives and since the curves
are nondecreasing (1) will be ensured optimal. The proof for
this is the proof of Theorem 2 in the appendix and is analagous
to Theorem 2 of Chapter II.
Again, continuing with the induction hypothesis, i.e.,
assuming n (i,a,k), n=3,4,..., k has an optimal rule given
by (1), the enumeration procedure becomes more difficult but
once again Theorem 2 of Chapter II follows in precisely the
same manner (this once again is so because alternatives only
have to be compared two at a time). This is done first, by
*
providing the analog of Lemrma 3 (proving ik+1l k ) and
secondly, by using the transitive relations that if ab is
nondecreasing and bc is nondecreasing, then ac is nondecreas
ing (ac=(ab)+(bc)). The details are in Appendix II.
Also, the discussion on piecewise convexity is simply
extended. The definition allows a function to be the smallest
of a denumerable number of convex functions and, hence, the
addition of the extra alternatives still permits n(i,a,k)
to be piecewise convex. Also, the argument related to the
terminal points holds since having the possibility of s
servers just allows for the possibility of s different weights.
This completes the discussion for the finite horizon and
we have shown that
Theorem 2: For an arbitrary number of servers our optimal policy
for the cost functions given in (2) is satisfied by (1).
Infinite Horizon and Averaging
Proceeding as before we shall prove that (1) is the opti
mal policy for the infinite horizon, i.e., when
(i) + BK +a p p (j. ,)
1
K(i)+B(kl)+C+:a 1 p $(j,a,l)
k .
*(i,a,k)= K(i) + kC +a p j (j,a,k) (4)
k+l
K(i) + (k+l)C+A+a L p k 1 (j ,ak+l)
ij
K(i) + sC + (sk)A+cr Z Pp (j,,s)
ij
Again the proof will be done for only one state (i,k) but by
its nature is seen to hold for all states. The proof is
Theorem 3 in Appendix II.
Moving into the discussion on the averaging criterion,
the assumption that for each number of open servers there are
only a finite number of decision points we are able to prove
that (1) is the optical policy as shown in Theorem 4 from
Chapter II. Theorem 4 in Appendix II gives this proof implying
that (1) is the optimal rule for t(i,k) where
T(i,k) = lim (la )4(i,as,k) where
T
1 T
W(i,k) = lim i Z g(i,k) and g(:t,k)is the one step loss
To t=l
function.
Concluding Remarks
Certainly, the extension of Theorem 5 in Chapter 2 holds
for the multiserver model. That is, we will never close all
the servers with customers present, except possibly when there
is a finite horizon.
In essence, therefore, this chapter along with Appendix II
shows that the multiserver and single server model are very
similar, in the sense of choosing optimal policies. This is,
however, an exception rather than the rule in dynamic program
ming formulations. Bellman points out that under very general
conditions when there are only two possible decisions that
there is some point which uniquely separates our decision region.
However, for more than two decisions he maintains that in general
our decision regions are not so divided.[2]
Bellman, R., op.cit., page 75.
CHAPTER V
CONCLUSION AND EXTENSIONS
The purpose of this paper was to ascertain optimal poli
cies for queuing systems. The form of the optimal policy for
the single server had been found before, but only when obser
vations were at the end of arrival and departure epochs, rather
than having observations in equal time intervals. Also, the
only work previously done in existence of optimal policies
for multiserver models was a twoserver case discussed by
Heyman. In this discussion, he always had one server open
and the policy was to determine when to open the other. We
allow for any finite number of servers. We are able to con
clude that the number of servers open at some point in time
is given by an intuitively appealing rule. This rule is
analagous to the (s,S) policy in inventory theory.
The applications of the work in this paper are quite wide
spread. In any situation where there is a variable number
of service stations which may be open, one would know how many
servers should be opened or closed. More might be learned
by observing additional computational results, as the purpose
of this paper is not to find optimal policies but only the
form at which they take on. Sobel points out many specific
problems relating optimal policies for queuing systems to
inventory and replacement theory.
There are several extensions to be seen from the work
done in this paper. Certainly there is a need for greater
computational facility in obtaining the optimal policy for
a given model. At present the closest computational experi
ence for the multiserver model is in a paper by Moder and
Phillips.2 Also, there is no reason why Howard's scheme
cannot be extended to deal with this case. Computational
experience is very important for a problem of this type as
its application will only become meaningful when the param
eters which we might be free to choose are set at the proper
levels. That is to say that even though our optimal policy
is determined only from the number in the system and the
number of servers open, there may be other parameters, such
as service rate, which might substantially affect our optimal
policy.
The main restriction that had to be made, so that periodic
review can be used, was that our arrival and service distribu
tions were exponential. As was mentioned this assumption could
be dropped if we are willing to increase our state space.
That is, the state would be not just a number in the system
and the number of open servers but also the time from the last
arrival, if arrivals were to become general. Investigation of
this problem is to be considered. Computationally, one would
almost certainly have to use Jewell's Markovrenewal technique.
Another extension is to consider finding optimal policies
when there are several machines with different service rates.
This was avoided because one must either make a predetermined
rule, or include the finding of the rule as part of finding '
the optimal policy to determine which of the available machines
an arriving customer will be serviced on. When this is done
one can introduce switching type policies as alternatives,
i.e., if a server is closed down while serving a customer,
the customer may immediately switch to another machine. Heyman
discusses rules of this type.
A slightly different but very interesting problem can be
thought of as an extension is that of finding the optimal
policy as a function of some other parameter. In particular,
finding an optimal rule as a function of the arrival rate,
rather than of the number in the system, may have useful appli
cations. If one could check on the rate at which customers
entered a supermarket, for example, it might be possible to
predict when servers should be opened or closed so as to avcid
idle time and long queues.
Of course, balking or reneging can also be introduced to
the system. In connection with this we could consider a modi
fied loss system which has application to telephone switchboards.
That is, consider a customer who tries to enter the system and
is turned away (placing a call and finding all circuits busy)
but will return with a certain probability. Associating a
cost with a lost customer it is to be determined how many ser
vers should be kept ooen so as to minimize cost.
Another change in the model would be to change the queue
discipline; introduction of priorities may certainly make this
work applicable to timeshared cor. puter systems.
APPENDIX I
PIECEWISE CONVEXITY
18
The following is due to Zangwill
Definition: Piecewise convex function (PCF)  A function
f(Z) is said to be piecewise convex if there is a
sequence of convex functions {gi(Z)} such that
f(Z) = inf {gi(Z)
i
Definition: Let f(Z) = inf {g.(Z)} be PCF. Define a basic
i
set B. as
1
Bi = {ZI f(Z) = gi(Z)}
Definition: Let the terminal set T be defined as
T = {ZI f(Z) = gi(Z) = gk(Z) for i / k}
The next lemma is an important one in the concept of PCF.
Lemma 1: Let {fi(Z)} be a sequence of PCF each with respec
tive terminal set T.. Then F(Z) = w.f. (Z), with
w. > 0 is PCF with terminal set T = U T..
Proof: The proof is by induction on the number of functions.
For one,the result is clearly true. Assume it is true
for k, i.e.,
k k k
Fk(Z) = w.f. (Z) is PCF with terminal set T = U T.
k I 1 i=l
i=1
57
By the definition of PCF
1
Fk(Z) = inf {g (Z)}
i
for some sequence of convex functions.
2
Let w k+if (Z) = inf{g (Z)} for a sequence of convex
functions. Then
1 2
Fk+ (Z)=FkZ) k+ f (Z)= inf{g (Z)}+inf{g (Z)
i j
1 2
= inf{gi (Z) + g (Z)}
i j
1 2
But gi (Z) + gj (Z) is convex for all i and j and hence
F k+(Z) is PCF.
Now, a point h is in the terminal set of gl () + g () if
1 2 2 1
and only if g (h) + g (h) = g (h)+g2 (h)=g. (h) and
1 g 12
either (a) il/i2 and jl=j2, (b) il=i2 and ji/j2, or
(c) i 2 and j.2. If (a) occurs then h e Tk, if (b)
1 ii and jdJ2.
occurs then h Tk+1 and if (c) occurs then h E T kTk+l.
k
Also, if h E T then either (a) or (c) occurs; if h e Tk+l
k+l
then either (b) or (c) occurs and h e T Another
useful lemma is,
Lemma 2: If g(Z) and h(Z) are PCF then f(Z) = min [g(Z),h(Z)]
is PCF.
Proof: By definition of PCF
g(Z) = inf g. (Z) } and h(Z) = inf {h. (Z)}
i 1j
for some sequence g and h.
Hence,
f(Z) = min
[inf (gi(Z)}, inf {h. (Z)}]
i j J
= min [inf (g.(Z) hj (Z)}]
ij J
= inf
ij
(gi(Z) h (Z) }
and f(Z) is PCF.
Lemma 3:
Spij.K(j)
j=0
and
E q..K(j) are convex.
j=0
Proof: (Two ways are
illustrated  one for p.ijK(j) and
1J
the other for F qijK(j))
E PijK(j) = E K(i+x)
j=0 'L x
= Pr (x arrivals in time period) K(i+x)
x
which is a linear combination of convex functions and
hence is convex.
Or as an alternative,
6K(i1+xy)+(le)K(i2+xy)K(eil+ (16)i 2+xy)
6E E K(i +xy)+(l9)E.E K (i +xy)::E E F(i +(i)i +xy)
x y X Y 2 X y 1 2
and hence
Sqij K(j) = E E K(i+xy)
j=0 x
is convex.
Recalling that,
0
9 (i,a) = min
1
N
K(i) + a Z p._ K(j)
j=0
K(i) + A + C + a q K(j)
j=0 ij
0
and using the definition of PCF we conclude that 4, is PCF.
Next, since
0 0 0
pi l (j,a) = (i+x, ) = E Pr(x arrivals) i (i+x,a)
j=0
and
E q
S~ 1 (j,a) = E E (i+xy,a)
j=0
0
= I Pr(x arrivals) Pr(y services)41 (i+xy,a)
along with Lemma 1 enables us to say that I pi 4. (j,a) and
Z qij 0(j,a) are PCF. Since the addition of a convex function
to a piecewise convex function is obviously piecewise convex,
2 (i,a) is a PCF, a result achieved from Lemma 2.
Finally, alternate use of Lemmas 1 and 2 leads to the
following:
Lemma 4: 0 (i,a) for all n > 1 is PCF.
n
The result also follows immediately for C (i,a).
n
Clearly, since the set of terminal points does not depend
on the weights,
O 0
SPi+xj 1 (ja) and pi+xyj k1 (ja)
k 1 i+xy kj ki
have the same terminal points. Also, each interval between any
two terminal points of a piecewise convex function is convex.
Hence, to show
[0 P0j 0a
[Pi+xj 1(j') Pi+xy,j 'k1 a)]
is nondecreasing between terminal points one can resort to the
discussion used to show the same thing for
E [pij qij] K(j).
60
Repeated use of the fact that the terminal points are
the same and that the expression is nondecreasing between
these points enables us to conclude,
0
Lemma 5: Z [pi+x, P i+x,j] tki (j,a) is nondecreasing
in i. Host certainly the result also holds for
Sqi+x,j qixy,j k1 (ja)
APPENDIX II
EXTENSIONS OF THEOREMS TO MULTISERVER QUEUE
Go
Lemma: Z [p pi K(j) is nondecreasing for m
j=0
Proof: Just as q. K(j) =
E j K(j) = E E ...E
3ij x yl
Letting
E E K(i+xy),
x y
K(i+x yj)
m j=l j
Sy. = Y from Chapter II we know that
j=0
K(i+xY ) K(i+xy ) for Y >Ym is nondecreasing
in i. Hence, so would the difference in their expecta
tions be and
m
E [Pi
j=0 i
n
Pij] K(j)
ij
for all m
Theorem 1: For n=l the optimal policy is of form (1) in Chap
ter IV.
Proof: Let k be the number of open servers.
Then,
K(i)+Bk + P p. K(j)
ij
1
K(i)+B(kl)+ C+c p. K(j)
ij
p1(i,k,a) = min K(i)+ kC+a T p.k K(j)
14
K(i)+A +(k+l)C+ac p. K(j)
K(i)+A(sk)+ CL E p s K(j)
1 
The policy (1) implies that there exist unique numbers
which determines exactly how many servers should be open
with i customers in the system, given k were open. Now,
for any two alternatives m and n, m
K(i) + (Constant) + a L p. K(j) (1)
K(i) + (Constant)' +c ~ Pi K(j)
or
m n
E [p p j] K(j) = Constant
can occur for at most one value of i. Since the left
side is nondecreasing from the preceding lemma, once
alternative n is chosen, alternative m can never be
chosen again for all nm
nondecreasing, which determines when an alternative should
be chosen.
kI k
1 2
Lemma: [Pij, (jc,kl) Pi t (j,',k2) is nondecreasing
for kl
Proof: This can be written as
E[1l(i+xYkla,kl ) l (i+xYk2 ,,k2)]
where the expectation is taken over random variables x,
Y1 Yk2. If the bracket is nondecreasing for all i then
certainly the expectation will be. So let us consider
Sl(i+xYkl ,a,kl) ~(i+xYk2 a,k2)
Now for any i,
i (i+xYkl ,,k 0 = K(i+xYkl) + (Constant)
1 1 k 1
+a L Pi+x_y ,K(j)
t"'Pi+xY,
1
The difference is nondecreasing in i since K(i+xYl) 
K(i+xYk ) is nondecreasing and the difference of the
2
summations is by the first lemma in the appendix.
Assuming that n (i,a,k) n=2,3,4,...r has an optimal rule
given by (1) and that
k k
S[Pij n (j','kl) Pi n (j,'',k 0] for kl < k2
is nondecreasing in i, (1) must be shown optimal for n=r+l.
Theorem 2: (1) is optimal for r
Proof: Since
K(i)+kB +a p U i tr(i,a,0)
ij *r
4 + (i,a,k) = K(i)+ kC+ac pi k (i,a,k)
r+L ij r
K(i)+(sk)A+sC+. Z ps (ias)
I j r
for optimal solution to be (1) it must again be true
for alternatives m and n, m < n, the difference must be
nondecreasing. This will guarantee that the sequences
(j*} and (i*} will be nondecreasing. Since this difference
is
[Pm 4 (iA,m) pinj (i,a,n)]+(constant)
ij r ij r
it is nondecreasing by the induction assumption.
Theorem 3: 'i(i,a,k), as defined by (4) in Chapter IV, has an
optimal policy given by (1) in Chapter IV.
Proof: (Done for an arbitrary state (i,k))
Define T. (_l ) as the cost associated with the jth alterna
tive j=0,l,...,s. For example,
T0 0n) =n K(i) + Bk +au pPij 'tn(ja0)
Then,
4 (i,a,k) = min T. ( n )
n j=O ,. nl
j=0,...,s
Let j(n) = index which gives the minimum of T. (n ), j=0,...,s.
This implies that
n (i,a,k) = T j(nl) (' n ) < Tj ((n1)
and
'p (i,a,k) = T. (' ) < T. (' )
n+1(i,,k) = Tj(n) ) n T j(nl) n)
Subtracting,
T (i,'a,k) n (i,a,k) max [ Tj ( )T (n( ) ,n '
ITj (n) n)Tj (n) (n ]
65
j=0 . .,
0
(K(i)+kB+a E p.. 9 (j,a,0)) ,
k
K(i)+Ck+a pZ k _(j,(a,k) 
IK(i)+Ck+a Z pi n
(K(i)+Ck+a E 4P (j,a,k)) ...
ij n
IK(i)+sC+(sk)A+a 7 pi n (jas) 
ij n1
(K(i)+sC+(sk)A+a 7 pi s (j,cs))l ]
r
< aZ p k j (j,a,k) ; (j,a,k)l
S ijj nn n
Hence
max 4I (i,a,k) ( (i, a,k) <
k
aEp maxi (i,a,k) ,
ij i n1
S n n
I 1
Define, U n max [ .;.n (i,a,k) .
and hence
2
U < aU < a U <
n+1 n ni
and,since
n U1
U < aU =
=0 n+l 1 ia
n=0 n=0
we conclude that
z [ n+l (i,a,k) p (i,a,k) ]
n=0
(i a ,k)
n
,k)
n1(i,a ,k)
anu
1
converges uniformly and that
n (i,a,k) a(i,a,k)
where (i,a,k) has been previously defined.
Theorem 4: (1) of Chapter IV is optimal for the averaging
cost criterion.
Proof: Let (jO' jl' '" jkl' ik+l' is) Pa
be the optimal policy for Y (i,a,k). Let fa t, with
lim a =1 be a sequence such that P = P for all t.
St a
Since the number of possible policies is finite such a
sequence and, therefore, P* must exist. Let P be any
other policy. Then certainly
p(i,(i,atk)> ,(iatlk)
Hence,
lim (1at) (i,at,k) > lim (1at) 4(i,at,k)
toC tO
and again using the result that
(i,k) = lim (1at) 4 (i, t,k)
tm
where a(i,k) has been previously defined, it is the mini
mum averaging cost.
BIBLIOGRAPHY
1. Arrow, Karlin and Scarf, Studies in Mathematical Theory
of Inventory and Production, Stanford University Press,
1958.
2. Bellman, R., Dynamic Programming, Princeton University
Press, 1957.
3. Blackwell, D., "Discounted Dynamic Programming," Annals
of Mathematical Statistics 36, 1965, 226235.
4. Cox and Smith, "On the Superposition of Renewal Processes,"
Biometrika 41, 1954.
5. Derman, C., "On Optimal Replacement Rules When Changes
of State are Markovian," Mathematical Ootimization Tech
niques, University of California Press, 1963, 201210.
6. Edie, L., "Traffic Delays at Toll Booths," Operations
Research 2, 1954, 107138.
7. Hadley and Whitin, Analysis of Inventory Systems, Prentice
Hall, 1963.
8. Heyman, D. P., "Optimal Operating Policies for Stochastic
Service Systems," University of California, 1966.
9. Howard, R., Cvnamic Programming and Markov Processes, MIT
Press, 1960.
10. Jewell, W., "Markov Renewal Programming I and II, "Opera
tions Research II, 1963, 938971.
11. Magazine, M., "Markovian Sequential Decision Processes,"
THEMIS Working Paper, 1968.
12. Moder, J. J. and C. R. Phillips, "Queuing with Fixed and
Variable Channels," Operations Research 10, 1962, 218231.
13. Nemhauser, G., Introduction to Dynamic Programming, J. Wiley,
1966.
14. Prabhu, N. U., Queues and Inventories, J. Wiley, 1965.
15. Sobel, M., "Optimal AverageCost Policy for a Queue with
Startup and Shutdown Costs," Operations Research 17,
1969, 145162.
16. Veinott, A., "On Finding Optimal Policies in Discrete
Dynamic Programming With No Discounting," The Annals
of Mathematical Statistics 37, 1966, 12841294.
17. Yadin and Naor, "Queueing Systems with a Removable Ser
vice Station," Operations Research Quarterly 14, 1964,
393405.
18. Zangwill, W., "The Piecewise Concave Function", Manage
ment Science 13, 1967, 900912.
19. Little, J. D. C., "A Proof of the Queueing Formula:
L = XW", Operations Research 9, 1961, 383387.
BIOGRAPHICAL SKETCH
Michael Jay Magazine was born in New York City on April 29,
1943. He was educated in New York elementary schools and grad
uated from Forest Hills High School in June, 1960. In June,
1964, he received the Bachelor of Science degree in Mathemat
ics from the City College of New York. In January, 1966, he
received the degree of Master of Science in Operations Research
from New York University. While at New York University he
worked as a graduate assistant in the Department of Industrial
Engineering and Operations Research. Since September, 1966,
Michael Magazine has been at the University of Florida pursuing
the degree of Doctor of Philosophy. During this time he worked
first as a research assistant and then as a research associate
doing research and teaching courses in the area of Operations
Research. In June, 1969, he was awarded the Master of Engi
neering degree from the University of Florida.
Michael LMagazine is a member of the Operations Research
Society of America, The Institute of Management Sciences, as
well as the honorary societies Alpha Pi Mu and Sigma Xi. He
is married to the former Joan Nemhauser and has one son Rcger
Eric.
This dissertation was prepared under the direction of
the chairman of the candidate's supervisory committee and has
been approved by all members of that committee. It was sub
mitted to the Dean of the College of Engineering and to the
Graduate Council, and was approved as partial fulfillment of
the requirements for the degree of Doctor of Philosophy.
August, 1969
Dean, College of Engineeri
Dean, Graduate School
Supervis r Committee:
h harrman
_ /
r

