Group Title: Affordable housing issues
Title: Affordable housing issues ; vol. 12 no. 2
Full Citation
Permanent Link:
 Material Information
Title: Affordable housing issues ; vol. 12 no. 2
Series Title: Affordable housing issues
Physical Description: Serial
Language: English
Creator: Shimberg Center for Affordable Housing
Publisher: Shimberg Center for Affordable Housing
Place of Publication: Gainesville, Fla.
Publication Date: February 2002
 Record Information
Bibliographic ID: UF00087009
Volume ID: VID00013
Source Institution: University of Florida
Rights Management: All rights reserved by the source institution and holding location.


This item has the following downloads:

Newslet-Feb02 ( PDF )

Full Text






M.E. Rinker, Sr., School of Building Construction College of Architecture PO Box 115703, University of Florida, Gainesville, FL
32611-5703 TEL: (352) 392-7697 SUNCOM: 622-7697 FAX: (352) 392-4364 e-mail: AFFHSNG@NERVM.NERDC.UFL.EDU

Volume XII, Number 2

February 2002

When providers of Ilc, iJ ii housing receive praise for the number of homes that they produce in a
given period, the implication is that the occupants of the homes have an improved quality of life. Further,
these occupants are more involved in the community and they have an enhanced self-esteem. It is these
sociological changes, rather than the number of homes built, that are the important impacts or outcomes of
affordable housing delivery. It is important that we assess success by measuring the right thing.
Reprinted below is an article titled, "Getting at Impact: A Beginner's Guide" that was authored by Dr.
Paul Clements and that appeared in the September/October 2001 issue oJ \ti, ii, i.... ', the journal of afford-
able housing and community building. \i, li I .... is published six times a year by the National Housing
Institute located in Orange, NJ. Inquisitive readers may wish to visit the Institute's web site at where it is possible to subscribe to \ti /'i i .... and to view back issues.

In 1999 the Sacramento Mutual Housing Association
(SMHA) owned and operated 492 units of democrati-
cally controlled affordable housing. It had demo-
graphic data on its low-income residents, and manage-
ment data on unit turnover, budget variances, mainte-
nance, volunteer hours, training, and youth programs.
But when staff and residents got together to discuss
how things were going, they realized that they still
didn't know how well they were achieving what they
were trying to do enhance self-esteem, develop
community power in broader political processes, and
promote a sense of community, neighborhood security,
participation in neighborhood organizations, and links
across social differences.
Most community organizations face similar ques-
tions. In recent years, some funders have started
asking for reports on impacts or outcomes actual

benefits for participants during or after their involve-
ment with a program. Funders want to know what
changes their money is making, how people's lives are
different. Most community organizations want to
know this too: Is what we're doing worthwhile?
Answering such questions is tricky, and most organiza-
tions are not used to doing it. But creating an impact
measurement process for your organization is both
possible and useful.
Moving to impact assessment is a big shift. Commu-
nity development organizations are more accustomed
to tracking inputs and outputs: Did you spend the
money in the amounts and categories you said you
would? (Answer by line item.) How many housing
units did you build? How many people went through
the training program? This kind of reporting encour-
ages a counting mentality, as though getting bigger
numbers were important by itself.


H 0 S N

Changing what we measure and report on can have
a big influence on how we carry out our work. This
might sound like the tail wagging the dog, but the
words and concepts we use can frame the way we
think, and as time goes on they can even influence
what we see. Focusing only on inputs and outputs can
lead organizations to continue building housing when
it no longer supports their actual mission, for example,
or to miss some of the indirect effects that their devel-
opment programs are having.
On the other hand, specifying the changes the
program is trying to bring about in terms that make
sense for participants, agency staff, and other stake-
holders can be an occasion to build a stronger consen-
sus on the program strategy and make sure it fits the
organization's mission. This can also shift the balance
of power, as funders are given an agenda developed by
staff and participants. More important, it focuses
everyone's attention on the shared transformational
task. Then the reports you write support thinking and
action that build the program.
But measuring impacts is not simple. "A sense of
community," "neighborhood revitalization," "empow-
erment," and "self-esteem" are all tricky concepts to
put our fingers on. We often rely on informal assess-
ments, feelings, or anecdotes to assure ourselves and
funders that we are on the right track. There is value
in those informal assessments, and they should never
be skipped or ignored. But sometimes when we are
asked to identify program impacts we jump too quickly
to things that are easy to quantify, or we ask questions
so broad and vague ("On a scale of 1 to 5, how posi-
tively do you feel about yourself?") that we're not sure
what the answers really mean.
It is possible to create a good, formal impact mea-
surement system, however, and there are good reasons
to do so. This information anchors ongoing strategic
planning and helps the organization demonstrate
results to funders and other stakeholders. The
information-gathering process heightens awareness
of the topics under investigation, and outcome
information helps different stakeholders hold each
other accountable.
Doing this well will take about five percent of an
organization's human and financial resources. It will
probably take more than this in the year you establish
your system if you are starting from scratch, but you
should expect it to run on about five percent a year -
and this should be included in every grant request. If
you can't release these resources or if you run on a

small and tight budget, it is probably better to keep your
impact assessment informal, based on observations and
There are several steps to developing an impact
evaluation process for your organization. First, you
need to figure out what impacts you hope to achieve.
Impacts are the changes in people's conditions due to
program activities compared to what we would expect
without the program. The number of children immu-
nized is not an impact; the reduction in disease rates is.
The number of units you build is not an impact; higher
levels of housing stability, improved neighborhood
safety, and improved conditions for the families moving
into those units are.
Your "benefits picture," or list of desired impacts,
should be created with input from program staff,
participants, funders, and other key stakeholders. A
benefits picture for a housing program might include
reduced monthly housing payments, a greater sense of
security and self-esteem for residents, a heightened
sense of community, stronger property values, and less
crime for the whole neighborhood. It is important for
the list of desired impacts in your benefits picture to be
complete, since it provides the framework all the way
along: for selecting indicators, for seeing how they work
together, and for interpreting the information when the
evaluation system is in use.
Some benefits may be relatively easy to measure
directly, such as rising property values, voting rates, or
community use of public spaces. But many will not.
Most likely, some of the more important outcomes in
your benefits picture will be qualitative and subjective.
Sense of community or quality of leadership cannot be
directly quantified. People have different ideas about
the quality of a house, what self-esteem involves, or
what makes a neighborhood attractive.
To measure such intangible benefits, we need
indicators. Indicators provide consistent steps for
collecting data that tell us something about the phenom-
enon we are actually interested in. The percentage of
people below a certain income is a pretty good indicator
of poverty. The number of people who know more than
three of their neighbors can be an indicator of commu-
nity cohesion. We know the information from indica-
tors is not going to be perfect. But it can nonetheless
give an organization a good idea of how it is doing. An
impact evaluation system should include a range of
indicators that fits the benefits picture.
Stop Abusive Family Environments (SAFE) in
Welch, West Virginia, started out as a provider of

shelter and other temporary services to battered
women. In 1999 it decided that the next thing to do
was to help its clients get into affordable, stable,
permanent housing, and it began a housing develop-
ment program. It also decided to evaluate the
program in terms of the improvement in housing
conditions for the women it served.
The quality of a house involves many different
factors: physical structure, safety, furnishings, hygiene,
space, and social features. SAFE devised a point
system: Two or more major problems with plumbing
and electrical features or having no insulation would
count for three points. Not being allowed to keep a pet
if you wanted one would count for one point, as would
excessive noise. Not everyone would agree on which
of these things were more important, but clearly if you
moved out of a house that got 20 points and into one
that got only two or three it would be a big improve-
ment. Staff brainstormed about features that contribute
to the quality of a home, and then interviewed resi-
dents to improve and prioritize the list. When you
have worked out an indicator like this, you need to
review it with representatives of all your stakeholders
to ensure that it reflects their priorities and perspec-
tives and to secure their buy-in.
To help practitioners take the initiative in deter-
mining indicators that speak to their needs, the
Development Leadership Network (DLN), a profes-
sional association of community development
practitioners, organized The Success Measures
Project (SMP) in 1997. The goal was to establish a set
of practitioner-driven "success measures" that would
be relevant, comparable, and easy to use and really
get at the question of impact.
SMP focus groups first created generic benefits
pictures, focusing mainly on housing, economic
development, and community building programs.
Then they developed dozens of practical, user-friendly
outcome indicators. Twenty-eight organizations have
been field-testing these indicators, and many more are
using the first edition of the Success Measures Guide-
book in which they are described.
Several community development corporations in
Detroit, one of the field-test sites, are using SMP
indicators to measure the sense of community, the
visual attractiveness of their neighborhoods, neighbor-
hood security, and the extent to which basic needs are
met by local businesses. For each of these categories,
the Guidebook provides ideas about what to measure,
ways to get information, and how often to measure.
For example, for "sense of community" it suggests

using a survey to track residents' participation in
community life, the share of residents who know
names of neighbors in adjacent buildings, and resi-
dents' satisfaction. It also recommends keeping track
of the number of organized community activities in one
year. For neighborhood attractiveness, the Guidebook
suggests gathering a focus group of residents to
determine a substantial set of small indicators (number
of abandoned cars, amount of graffiti, condition of
sidewalks), and then doing a neighborhood tour to
assess these indicators block by block. It recommends
that these tours should happen at least once a year, and
warns groups that they will need three to four years of
data before trends can start to be identified. The
Guidebook provides similar information with varying
amounts of detail, for all its other categories of impacts.
Working from these starting points, the Detroit
groups have designed specific survey instruments.
They are available from DLN, with sample reports, as
models for other organizations.
Once you choose a set of indicators, check them back
against your benefits picture to see if you have left any
desired impacts out. If you have not found an indica-
tor for one or two of them, that may be all right. Some
things you care about may simply be too hard to
measure, or the effect of your program may be over-
whelmed by other factors. Still, it should be possible to
get a good representation of most of the more impor-
tant program benefits in your benefits picture. Make
especially sure that you are not focusing on factors that
are easier to measure but less important. Also, keep in
mind the results you are not measuring, so you can
continue to make informal judgments there.
The information from a survey or questionnaire can
show program outcomes but not impacts. Impacts are
the actual changes compared to what you would expect
without the program. To get at impacts you have to
interpret the data using judgment based on your
overall knowledge of the program and its environment.
In some cases it may be possible to make a comparison
to a similar location with no program like your own.
As years go by you will have a record of program
results that can inform planning, funding requests, and
communications about the program in general.
In the long run, programs that can demonstrate
their effectiveness are better positioned to grow and
to be replicated. As impact assessment becomes
more and more common it will make the community
development field more professional while also
strengthening links with communities. Government
offices and foundations will be better able to justify
investments in community development, and the
public at large will see more clearly the value of
community-based approaches.

Some Tips on Collecting Data

Once you have identified indicators, you need to
collect data on them on a regular basis so you can see
how they are changing. Indicators are only useful to
you if you can use them to show the before-and-after
impact of your work. Good principles for data collection
are to use the least resources to get the best information,
to include participants or community members as much
as possible, and to produce reports and integrate the
results back into your management and planning
processes as quickly as possible. Here are some tips.
An off-the-shelf database package like Access is usually
the most efficient way to input and manage the data,
but depending on the amount of information and
organizational resources, a standard spreadsheet
program such as Excel may suffice.
Surveys should be brief and to the point, with no more
questions than necessary.
The purpose of a survey should be transparent to
respondents from the questions alone. It should only
take a couple of minutes to explain the purpose of
the questionnaire to the respondent.

* Test survey questions on several people before
conducting a full survey to be sure that the people
administering it understand and are comfortable with it
and that respondents will interpret the questions the
way you intended.
* When you want to know the experience or opinions of the
community as a whole select respondents randomly. If
you just interview people known to staff members, for
example, you're leaving out those who have no
connection to your organization and their views and
experiences may be very different.
* Make sure the data collection strategy is reviewed by
someone who understands survey design.
* Integrate data collection into the normal flow of your
program. It often makes sense to administer a
questionnaire at the point at which people enter
program activities, such as when they are approved to
buy a house or when they begin training. Be sure to
explain the timing of any subsequent questionnaires
they will be expected to complete. Repeating the same
survey at later points in the program can provide a good
measure of program outcomes.

Affordable Housing ISSUES is prepared bi-monthly by the Shimberg Center for Affordable Housing for the purpose of
discussing contemporary issues facing affordable housing providers. Reproduction of this newsletter is both permitted and
encouraged. Comments or questions regarding the content are welcome and should be addressed to Robert C. Stroh, Director.

I 9ZC Id 'a0IAsaurIe
01 3d)V 'alnpa41ip4TV jo aJanIo3
uoqpn~qsuo uip!ing jo loot3s
u!snoH ajqp.Tojjv TojJ .aua:) aquinU{S


University of Florida Home Page
© 2004 - 2010 University of Florida George A. Smathers Libraries.
All rights reserved.

Acceptable Use, Copyright, and Disclaimer Statement
Last updated October 10, 2010 - - mvs