Sophia.org – A Collaborative and Accessible Online Learning Community
Alfonso Sintjago – Intro to Evaluation – Project B: Evaluation Context
Oct 21/2011
Object Summary
Sophia.org provides students with an online social learning environment, where students and instructors can develop relationships with one another, and contribute to a robust social network capable of rapid expansion and fueling innovation. The program, located physically in Minneapolis, is accessible from anywhere in the world where there is free access to the internet. Their community is everywhere. However, having primarily English resources, most of Sophia’s audience is in the United States, and more specifically, in the Midwest. Sophia has various programs including free access to community developed educational videos and resources that can be rated by anyone who is a part of the community, a learning management system which integrates social media, the use of groups, and forums to further explore concepts and ideas, and Sophia Pathways, a program with over 20,000 educational videos where an institution can purchase a computer enhanced math rapid assessment program. Sophia hopes to become a key provider of digital educational content, as well as a major player in the rapidly expanding market of computer enhanced rapid assessment programs.
Procedures Used to Gather Information
The majority of the information was gathered by visiting the Sophia.org website. Sophia.org regularly updates an online blog and has produced various web based interviews where their founder and CEO Don Smithmier explains the different functionalities and capabilities of Sophia. In addition, other resources were gathered during a site visit, where I met with the Vice President of Academic Outcomes, Dr. Angiers, and further discussed the functionalities, mission, and future of Sophia.
Purpose of the Evaluation
This evaluation primarily focuses on determining to what extent the program goals and objectives are being followed. While the program is currently in its beta stages, the website is already available to the public. As an online website attempting to construct a community, Sophia’s staff believes that through the use of site analytics as well as written or spoken feedback that they will be able to increasingly meet the demands of their users. In this way, Sophia will likely continue to be a developing model. Feedback is received any day during any time of the day. Since the website became accessible, there have been no attempts to measure or assess the program’s success in increasing students’ knowledge. This may be partly due to the early stage of the program and also because of the role it plays within informal learning. It is difficult to assess how much a particular website contributes to learning unless an evaluation of content retention is included within the online modules. Since Sophia disseminates information rather than evaluates student retention, it will be difficult to obtain a better idea as to how helpful the program is to students, apart from through data on page visits and feedback forms. Sophia currently has a limited number of resource packets. As a collaborative web platform, this program would benefit from increasing their number of contributors.
By evaluating site analytics data including the number of page views, length of stay in the site, and the level of interaction between participants, an evaluation of Sophia demonstrate the potential of the program and the possibilities for the platform. Evaluating the platform’s effectiveness is useful for furthering the impact of the program through marketing and the establishment of new partnerships. While Sophia.org is freely available to anyone with an internet connection, its platform helps to attract users to a fee-paying service, Sophia Pathways. Sophia Pathways then generates the revenue needed for the sustainability of the program. Through marketing and by offering a unique and aesthetically pleasing experience to users, learners will be more likely to come back. The competition and the market for informal online learning platforms is likely to remain volatile for years to come. Without growth and the constant acquisition of new customers, Sophia will likely be unable to sustain itself over time. The list of failed innovative online educational projects is extensive.
Learning from the CEO’s successful experience with Capella University, and having an understanding of the possibilities for future growth in the market, Sophia is well situated to play an increasing significant role within that emerging industry. Some scholars have quantified trends and projected a work environment where informal education increasingly plays a role. As more and more students participate in online courses, and supplement their education informally through the internet and other resources, the greater the user bandwidth and opportunity for Sophia will become. Sophia’s emphasis on the customization of modules and the possibilities for different instructors to explain the same subject in a limitless number of ways is another reason why they could greatly benefit from expanding. Being able to understand why its user base has expanded and how this expansion rate could be increased are two ways in which an evaluation of Sophia could benefit this innovative startup company.
Evaluation Stakeholders/Audiences and Their Concerns
Sophia’s CEO, Don Smithmier, would be the primary stakeholder of this evaluation. Smithmier is the founder of Sophia and it was because of his vision that the current Sophia executive board was assembled. After leaving his position as a VP for Capella University, he has devoted his interests and efforts to a number of innovative startup companies. Sophia is one of his most visible projects. The success or failure of Sophia will impact the perception of his management capabilities and innovative vision. The non-profit element of Sophia is not generating substantial revenue and, by remaining free from advertisements, without donations or the rapid growth of Sophia Pathways its sustainability is questionable. Because of this, Don Smithmier has conducted a broad marketing campaign and attended numerous events both online and in person. Don Smithmier has an understanding and familiarity with the stock market and has an interest in maintaining a positive image of the project to encourage further investment into his company.
Vice President of Academic Outcomes, Dr. Angie Eilers, focuses on the importance of providing both rigor and flexibility though the Sophia platform. Having written a number of articles regarding educational reform and currently advocating Sophia’s capabilities as an instructor and educational tutor, she would stand to lose some credibility as an education reformer if the project fails to improve its audience’s academic performance. Does Sophia help to make its users more prepared to succeed academically? Sophia’s uniqueness as a channel where individuals of different skill levels can find a video to learn from is an example of Sophia’s platform’s flexibility, yet having a relatively small number of videos, it is currently not possible for an individual to use Sophia as their only educational supplement.
Senior Vice President of Business Development, Allison Gate, has been a successful entrepreneur in a number of businesses, including transforming a business from a startup to having an over 600 percent increase in sales and eventually being acquired by GE health care. With extensive experience as a business leader in the private sector, the transition of Sophia’s learning platform into a financially successful company is her primary responsibility and concern. Functioning on a subscription rather than an advertising model, becoming a financially sustainable business is not directly linked to increasing its overall number of visitors. As businesses which offer free services have experienced, despite having an extensive and growing audience, it has been difficult for websites such as YouTube to generate a profit despite the use of advertising. Offering a similar service without the inclusion of ads, increasing the number of visitors may also not result in a financial profit for Sophia.
The Chief Operating Officer, Steve Anastasi, is primarily responsible for the development and improvement of the Sophia platform. He worked on developing the Best Buy e-commerce platform and has over 25 years of experience. His experience led him to develop a successful software company that was then acquired by a larger firm (but he continued to work for it as an IT director). If the Sophia platform fails to appeal to the general public because of the interface capabilities, or browser incompatibility problems, as well as glitches or bugs in the code of the site, Steven Anastasi would be most directly linked with these difficulties. With his ample experience Sophia hopes to develop a platform that is not only stable, but also attractive to internet users of different age groups.
The Vice President for Program Management, George Tattersfield, was a co-founder of Guaranteach, an adaptable learning environment where students are given new, more difficult math assignment at a pace that matches student improvements. The financial sustainability of Sophia partly depends on the appeal of Sophia Pathways which was developed primarily from content previously generated by Guaranteach. This includes over 20,000 educational videos. Apart from students’ possible educational gains from using Sophia’s freely available educational materials, Sophia Pathways’ personalized and adaptable learning program has been purchased by a number of schools and could be used by parents of homeschooled children to increase the quality of their children’s learning experience. Negative assessment of Sophia Pathways reflects negatively on the perception of Sophia as a whole, but also on the Vice President for Program Management, since he was involved in the development of this platform and has been in charge of adaptation to meet the needs of different educational institutions.
The website’s visitors, users, and content producers are also stakeholders. As volunteers or beneficiaries, they could migrate to a different online environment if their needs are not met by Sophia. Producers could share their educational videos through Youtube or a different channel if they felt it would be more beneficial. These stakeholders participate in the project because of its flexibility and accessibility. Sophia allows every learner to become a teacher, and every teacher to become a learner. If an evaluation of Sophia were to highlight the benefits of the platform to its users, particularly their benefits in terms of educational gains, Sophia may increasingly attract more users and instructors, enriching the appeal of the site to other future instructors and students. As websites go “viral”, and suddenly grow at a rapid rate, Sophia.org could quickly become a main destination for internet learners. A positive evaluation could have a major impact in the appeal of Sophia to internet users. An evaluation could also help highlight the ways in which Sophia is not meeting the needs of a particular type of online learner or the needs of instructors, acting as a way in which to have a better understanding of the variables that could be improved in order to be more appealing to some current, and possible future, users. The success of the free components of Sophia is probably the best way in which Sophia can increasingly become visible and then it will be more likely that its private services will be purchased by schools and learners. Many internet businesses function on the 1% or 2% model, where most of the users are free riders and only 1 or 2% pay for additional features. This model has resulted in a sustainable revenue stream for many businesses. It is possible that a similar percentage increase from non-users or public users to private users may be all that is needed to meet the sustainability of goals of Sophia.
Apart from these stakeholders, society in general and learning technologies programs would benefit greatly from learning from Sophia’s experience and to what extent it has captivated the attention and minds of its users. As a rapidly changing learning technologies environment, an analysis of the success or failure of programs over time can help us to better understand both the future of educational business through the internet, and the learning experience of internet users .If successful, Sophia could rapidly expand or similar sites could be developed. Much of this impact depends on whether Sophia is portrayed positively or negatively in the evaluation.
Primary Intended Users of the Evaluation
Don Smithmier, the CEO and founder of Sophia, would have the greatest opportunity to benefit from this evaluation. As an innovative CEO and having ample experience with DOT COM economics and education, Smithmier is aware of the constant change and improvements needed from internet business in order to remain relevant in a rapidly changing cyberspace. Having to change or modify certain aspects of Sophia which were not as positively evaluated as others would lead to long term improvements to the Sophia platform. Whether the evaluation is dissatisfactory to the main players or satisfactory, as a beta project Sophia can use these findings to further strengthen its program, and through these improvements, capture a larger audience and increase its sustainability. Outside of Sophia, the evaluation would be of interest to academics who are trying to better understand the future of online learning.
Evaluation Questions
The questions below have been written to address the concerns of Sophia’s stakeholders.
- How quickly is the number of users to the site growing? At what rate are visitors opening an account after visiting? At what rate do they return to the site after starting an account?
- Once a visitor has an account, how often do they visit other producers’ packages? How many packages do most producers make? Are producers and users asking for information about Sophia Pathways?
- To what extent are the users of Sophia Pathways satisfied? Are the students’ math scores improving faster through Sophia Pathways than without it, according to national and regional standardized tests? Are students expressing high levels of frustration when using the system?
- Where are most of the visitors arriving from? Is there a particular advertising system that is working more successfully than others? What types of resources do most of the visitors visit? Which site organizational tool are they more likely to use? Do they stay on a particular page for more than 10, 20, or 30 seconds?
- Do visitors visit various resources after landing on the site? Do they seem to find one that meets their expectations or do they leave the site soon after browsing and searching for a couple of times? Are some key words more successful than others? Are some resources being regularly searched for but are not currently available?
- When interviewing package producers or instructors, what are some of the reasons they create educational packages? What is their motivation? Are their expectations being met by Sophia?
- What are the costs of Sophia’s nonprofit program in comparison to the cost of Sophia Pathways? Which one has a higher user satisfaction rate? What percentage of Sophia Pathways partners are renewing their contracts?
- Are students and producers experiencing technical problems when using the site? This includes bad links, browser error messages, and difficulty using the site with a particular browser.
Proposed Role of the Evaluator
Sophia will be evaluated by an external evaluator. As an external evaluator, Alfonso Sintjago has extensive expertise in evaluating online learning sites. We are possibly currently experiencing a second DOT COM bubble. With this in mind, the evaluator will carefully analyze what Sophia offers in comparison to other competing programs. The evaluator does not have a stake in the outcome of the evaluation as he does not currently work for a competing company and is primarily interested in learning about the differences between these programs. He currently works for the University of Minnesota as an Instructional Technology Fellow and has a teaching background in Learning Management Systems (LMS). He is well versed in using various systems of analysis and categorization to identify what type of LMS would be most useful and user friendly for a particular population of learners and instructors.
Having a strong interest in the possibilities for improving education by creating educational videos, he has visited various sites that use similar delivery formats. His extensive knowledge of LMS and video enhanced education should aid the evaluator in writing a useful report about Sophia that could highlight both strengths and weaknesses. As an external evaluator, while he hopes Sophia is successful as an enterprise, he is not reliant on the support of Sophia’s leadership team and his employment situation will not be impacted by the content of the report. If a member of the leadership team loses their job, it would not have any impact on the evaluator’s ability to conduct the evaluation. This is one of the benefits of being an external evaluator, as the evaluator’s job security is not tied to the success of the organization.
Despite being aware of the limited use Sophia is currently receiving from educators throughout the United States, the evaluator is also aware of the possibilities of the platform and its innovative nature. Reading both negative and positive feedback, Sophia has become a useful virtual space in which educators can share their knowledge with their peers and all types of students. As an advocate of the capabilities of digital technologies to positively influence education, the evaluator’s general orientation toward the program is inclined to be positive, as he believes in the merits of the program’s infrastructure, goals, and objectives. Despite having a strong appreciation for both qualitative and quantitative data, most of the evaluator’s prior work was of a qualitative nature. Nevertheless, the evaluator is aware of the importance of analyzing the various statistical indicators that have been stored and categorized by the site analytics. However, in his evaluation, he will combine the use of focus groups and interviews with an analysis of the data collected when visitors arrived at the website and their actions were recorded by the system. The focus groups and interviews will allow the evaluator to have a more in-depth appreciation of the stakeholders’ conscious concerns. Despite the evaluator’s lack of experience in conducting formal program evaluations, he has completed college coursework in evaluation, statistics, focus groups, and other research methods. He has also completed a number of courses in the learning technologies department and has an extensive history of working with educational technologies and Learning Management Systems.
Constraints of the Study
The greatest difficulty in evaluating Sophia will be to find out why many of the individuals who visited Sophia on one occasion decided it would be better to not participate and create an account on the website. Most of the individuals who could be contacted for interviews, online focus groups, and surveys are those who are currently members and have shared their contact information online. However, a large number of visitors did not get to this stage and very few of them filled out the feedback form explaining the reasons why they were not planning on joining the community. Obtaining information from individuals that had an interest but quickly decided it was not what they expected would be helpful in understanding some of Sophia’s current deficiencies and decide whether some of them can, and should be, addressed.
With a small staff with a large responsibility, it is difficult to obtain access to Sophia’s employees to discuss in detail what they consider their roles to be and learn from them what they consider to be problems and successes of the program. Another possible limitation includes the reality (in contrast to the expectations) of how much analysis will be possible using their site analytics. While they current collect extensive usage data, some of which is displayed on every Sophia package webpage, and more detailed information is expected to have been collected by the site, it is uncertain how much of this data will be sortable and accessible to the evaluator. It is possible that some of this data may have been lost in a prior server update and the anticipated comparison of usage over time will be limited by the periods and quantity of data that are currently available. Another constraint will be the timeline of the evaluation. Despite the need for an extensive evaluation, the researcher will only have time to fully address some of these questions. Choices regarding what questions to pursue will have to be made, limiting the attention that can be given to other valuable and interesting questions. Depending on the data and the extent to which it was already pre-sorted by the computer system the statistical analysis will be easier or harder to complete satisfactorily. If other individuals volunteer or decide to contribute in other ways to the evaluation, a larger number of questions about Sophia may be successfully answered.
