In the last nine steps, I have walked through Are We There Yet? A Communications Evaluation Guide by the Communications Network. It started with figuring out what to evaluate and establishing an overarching goal. The central question was: how are your communications efforts creating a change; more specifically, a change in behavior. The central message was: measure outcomes, not outputs. It ended with figuring out specific tools and establishing a evaluation budget.

Evaluating communications in social media, focused on creating change, is one aspect. I have occasionally offered a brief perspective on evaluating marketing efforts, focused on sales and commerce. And while I wholeheartedly believe social media should be used for sales and commerce purposes—as long as you play by the new rules of social media—and that those efforts should be measured, the first and foremost reason for engaging in social media should come from a mission statement-inspired goal.

I hope the walk through was helpful and insightful. Perhaps you have suggestions or improvements. If so, don’t hesitate to let me know. Likewise, leave a comment if you have an evaluation story to share or if you have any questions on your own evaluation efforts. I’m looking forward to hearing from you.

Happy evaluating!

You should now be clear on what your communications efforts are expected to achieve. You know your goal, objectives, audience, baseline and evaluation questions; the next step is to develop specific metrics to answer your evaluation questions.

The authors introduce milestones at this point of the evaluation. If objectives are intermediate markers toward your goal, milestones demonstrate your progress toward those objectives. Just like objectives, they are not intended to show your activities, your outputs, but rather your results on your way to reach your objective.

Milestones can either be measured quantitatively or qualitatively, or of course, a combination of both. Here are some examples of milestones and some of their possible metrics and measurements put into the context of familiar objectives (keep in mind, the metrics measure the milestone, not the objective per se):

Communications

Objective
“By the end of year one, have established an online community of 25,000 active, engaging members.”
Milestone
“By the end of month six, have solicited one action from 10,000 community members”
Quantitative/qualitative measurements
You have established a rate of interaction of 10% of your total Facebook fans.
You have established a fan base of 20,000 members on Facebook.
…

Objective
“Position your organization as an essential element of your arts community by the end of year two.”
Milestone
“Establish a relationship with bloggers from the 50 most-read arts blogs by the end of year one.”
Quantitative/qualitative measurements
Communication exchanges indicate increased interest by bloggers/creators.
Organization is referenced positively in at least five key blogs.
…

Marketing

Objective
“By the end of year two, increase annual revenue from online referrals from Facebook and Twitter by $50,000.”
Milestone
“By the end of year one, have established Web traffic of 150,000 annual visitors referred from Facebook and Twitter.”
Quantitative/qualitative measurements
You have established a fan base of 100,000 members on Facebook, with an average click-through rate of a posted item on Facebook at 5%
You have established a fan base of 200,000 members on Twitter, with an average click-through rate of a posted item on Twitter at 2%
…

The authors advise to identify three milestones per objective and three metrics or measures per milestone. They advise against tracking too much data; only track what measures whether you are reaching your milestone.

As we have seen in the baseline step, there are many tools out there to collect the data you need. Step eight will delve a little deeper into those tools.

On Monday, we’ll look at step eight: selecting your evaluation techniques.

(Source: Are We There Yet? A Communications Evaluation Guide)

The first question before posing your evaluation questions is: what stage is your communications effort in? The authors make a distinction between early, mid and advanced stages.

Early Stage

The early stage means the development or early implementation of a communications plan, where there is a degree of trial and error. The evaluation at this point helps determine the larger strategic direction.

Questions include: have I tested my messages with a sample group that is representative? Where have my messages first appeared? Are my messages right for the channels I selected? How is the audience responding to my tactics?

This is where a lot of orchestras and classical music organizations are now, whether they have implemented a communications plan or not. A lot of trial and error and determining whether the messages or working. Many organizations have gone about this without a communications plan and have found that the usual marketing and public relations messages don’t translate well in social media. They might get some responses from the audience, but nothing more than a comment or a “like” of their Facebook status. With a clear communications plan, this should get better, and those are the questions to ask.

Mid Stage

The mid course stage means you have a well-established communications plan, identified key influencers and players and have chosen to invest in certain strategies and tactics. The evaluation at this points helps determine whether you are on track toward your objectives.

Questions include: is my audience more engaged with the issue? Are bloggers and creators engaging with the organization? Do I need to change my message or messengers? Are there any environmental changes that require adjustments to objectives and tactics?

Not many organizations will be in this stage. Fewer have gotten to this stage with a clear communications plan, and even fewer will ask these evaluation questions. This should be the point where you can see some early signs of impact, if you are doing it right. Perhaps you find that you need to change your messages or messenger? Perhaps you need to shift away from Friendster because the environment has changed?

Advanced Stage

The advanced stage means you have made several measurable steps toward reaching your goal, but the environment might have seen some changes since starting your communications efforts. The evaluation at this point helps to determine what lessons you have learned from earlier steps and provides an opportunity to sharpen the goal and direction of your plan.

Questions include: are there signs of change in intention or behavior? What have I learned from achieved objectives and what is the organization’s role in the future? Does the data indicate a need to change my strategy or tactics?

Have your supporters started to organize? Are they putting the network in action? You should now be able to have some definite signs that your communications plan is doing its job. On the way, you will have learned some valuable lessons that you can apply to your future efforts and direction. Has your organization’s role changed on your road to achieve your goal? If so, what has changed and why?

And of course, can you claim to have aided and contributed toward reaching your goal? How much of it was due to your communications effort as opposed to other factors? And lastly, you will want to share your findings with colleagues, the board of directors, and even others in the industry.

Tomorrow, we’ll look at step seven: drafting your measurements.

(Source: Are We There Yet? A Communications Evaluation Guide)

In order to evaluate your results, you have to know your starting point. You will have to compare your baseline data with the new data you are going to gather as you progress toward your goal. You need exactly the kind of baseline data that will be able to show you whether your communications are working; and your goal and objectives will tell you what kind of information you need.

Looking at the goal and objective we discussed earlier, you would need data that tells you what you need to know about your network of supporters and your online community, and how engaged they are.

Many orchestras have detailed information from years of analyzing ticket sales data, audience surveys and of course Web site analytics programs. The League of American Orchestras and the National Endowment for the Arts have libraries of data as well. This data can help you in establishing your baseline, but remember; those numbers will need to be able to tell you something about your progress toward your goal and objectives. Use the data to answer questions such as:

• What is the awareness of and attitude toward your organization (or classical music in general)?
• What are some of the misconceptions and misinformation about your organization (or classical music in general)?
• Where are other classical music and arts organizations? How do they compete for your audience and online attention?
• How has your organization (or classical music in general) been presented in social media?
• Who are the influencers/creators that support or oppose your organization or mission?

These questions remind me of Kami Huyse’s Triad of Measurement that I mentioned in my e-book. The three measurement elements include: interest (how interested are people in your organization/issue); attitude (what attitudes do people hold about your organization/issue); action (what actions do people take as a result of your communications).

Looking at the baseline data in marketing terms, you will need to establish your current Web site sales data (e.g. average order size, churn rate, acquisition cost etc.), Web site traffic data, and perhaps most importantly, part of those two elements, your conversion rate (e.g. how many people driven to your Web site by Facebook or Twitter ultimately buy a ticket, recording, or take some other defined commerce action).

Tomorrow, we’ll look at step six: posing your evaluation questions.

(Source: Are We There Yet? A Communications Evaluation Guide)

Organizations often take the easy way out and measure what’s easy to count, such as visitor numbers for their Web sites. But those numbers won’t tell you what those visitors did with the information or content they found. According to the authors of the guide, “you’re ultimately trying to move an audience; therefore, getting feedback from the right source is crucial for a good evaluation.”

The authors tell us to be specific when identifying the audience. Simply the “general public” or “young people” (a particular favorite among managers that jump in with social media efforts) is not acceptable. Narrow your audience to a well-defined group. But identifying your audience does not simply mean knowing, for example, the demographics of those who are most likely to visit a performance (white, 55-64, household income of…). It’s all about behaviors.

Marketing expert Doug Dome explained to me: “behavior and financial value of behavior are at the heart of integrated marketing communications.” The financial value of behavior is an important aspect for your commerce goals, but there is more value to behavior than purely money. And that other value is what we’re trying to evaluate here.

In our example, you will want to evaluate those online users who are most likely to influence others, those who will become your evangelists.

The authors of the book “Groundswell: Winning in a World Transformed by Social Technologies” created so-called Social Technographics, a classification of people according to how they use social technologies. Here are the different categories and its public’s behaviors:

Creators
Publish blogs, Web pages, stories, audio/music/video

Critics
Post ratings, reviews; comment on blogs/forums; contribute to/edit wikis

Collectors
RSS feeds, add tags to pages or photos, vote for web sites

Joiners
Maintain profile on social networking site; visit social networking sites

Spectators
Read blogs, watch video from others, listen to podcasts, read forums, ratings and reviews

Inactives

In social media, the creators and critics will be your audience. Those are the people that are most likely to create content that advances your mission, content that contributes to your goal and objectives. Their online behavior influences others. But keep in mind; you have to concentrate on the relationships, not the technologies. It’s about what they do, not necessarily about what they use.

In addition, the authors of the guide advise that your evaluation should include both the audience targeted for your activities as well as any other audience who may influence your goals and outcomes. How are the collectors, joiners and spectators contributing toward your goal?

It would, for example, be interesting to learn how they have made their purchasing decision. Were they influenced by the creators and critics, your target audience?

Tomorrow, we’ll look at step five: establishing your baseline.

(Source: Are We There Yet? A Communications Evaluation Guide)