Frequently Asked Questions
Frequently Asked Question Topics
Top 5 Most Frequently Asked Questions
-
How much will it cost to participate in the VALUE Institute?
The work of VALUE has evolved from standalone rubric tools to demonstrating the proof-of-concept through the three Collaboratives, and is now poised to transition to a self-sustaining Institute. Cost for assessment of student work artifacts through the VALUE Institute is $6,000 to assess 100 pieces of authentic student work. This fee covers the expenses to:
- Assist with developing a sampling framework and strategy
- Upload student work artifacts in the Institute database (Watermark/Tk20/LiveText platform – you do not have to currently use these platforms to participate)
- Address one learning outcome (VALUE rubric) across each rubric dimension for 100 pieces of student work
- Have all artifacts double-scored by certified VALUE rubric scorers
- Generate scoring results reports, report templates, as well as the raw institutional data from scoring
The fee for each additional outcome (and 100 artifacts) is $4,000
-
Who is the VALUE Institute designed to serve? What types of institutions should participate?
Participation in the VALUE Institute is open to all types of higher education institutions/providers.
-
Is participation limited to AAC&U member institutions?
No. AAC&U membership is not a requirement for participation in the VALUE Institute.
-
Can we participate in the VALUE Institute as a consortium?
Any individual campus or program within a consortium may participate, and campuses may indeed choose to participate as a consortium in the VALUE Institute for purposes of data sharing, resource sharing, etc. Additionally, there may be opportunities to engage with such groups as the Multi-State Collaborative (MSC), an on-going collaborative partnership between AAC&U and SHEEO, for two- and four-year public institutions. Institute staff are willing to assist institutions in thinking through possible consortia arrangements.
ABOUT
What is the VALUE Institute?
The VALUE Institute is a continuing resource for higher education institutions to document, report, and use learning outcomes evidence to improve student success in college. Developed as a partnership between the Association of American Colleges and Universities (AAC&U), Indiana University's Center for Postsecondary Research (IUCPR), the State Higher Education Executive Officers Association (SHEEO), and our technology partner, Tasktream/TK20/Livetext, the VALUE Institute enables any higher education institution to utilize the VALUE rubrics approach to assessment by collecting and uploading samples of student work to a digital repository and have the work scored by certified VALUE Institute faculty scorers for external validation of institutional learning assessment. Participating institutions receive data and reports from the tested VALUE nationwide database for benchmarking student learning.
The VALUE Institute represents a systems-level innovation that draws upon decades' worth of fieldwork and empirical research that has shaped the VALUE rubrics' evolution from a set of tools implemented locally on individual campuses (2008-2014) to a powerful model for inter-institutional, inter-sector, and inter-state assessments of student learning (2014-2017). With no other existing available model for this important work, the VALUE Institute is uniquely positioned to provide evidence that allows faculty, institutional leaders, accreditors, policy makers – and even students themselves – to assess and improve the levels of achievement on a set of cross-cutting competencies important for all disciplines.
What is VALUE?
An acronym for the Valid Assessment of Learning in Undergraduate Education, VALUE is an approach to assessing student learning. This approach draws evidence from actual learning artifacts (papers and assignments) produced by students demonstrating their achievement of specific learning outcomes versus utilizing a standardized test divorced from the curriculum.
What are the VALUE rubrics? What is the VALUE approach to assessment?
Created by teams of faculty and co-curricular professionals in 2008 and released in 2009, the VALUE rubrics answer the need for measuring the development and application of the essential learning outcomes that college graduates need to be productive in work and in citizenship over time. The VALUE rubrics are available for download – at no cost – on the AAC&U website (https://www.aacu.org/value).
VALUE is grounded in research and best practices derived from the learning sciences— including educational psychology, cognitive psychology, student development theory, and instructional design—and generates robust data that lends itself to both qualitative and quantitative methodological consideration. Much more than a collection of rubrics, the VALUE process is a triad comprised of (1) the rubrics that describe each learning outcome on which student work will be scored, (2) the faculty trained as scorers who use their expert judgment to evaluate student work products and assign a score based on the rubric dimensions and performance levels, and (3) the student work products generated in response to a faculty-designed assignment from an actual college course.
Is the VALUE Institute like other AAC&U Institutes?
At present, the VALUE Institute is NOT an in-residence, team-based institute like AAC&U's other summer institutes (https://www.aacu.org/events/summer-institute), but rather a place where student work can be uploaded and scored for quality. Having said that, the Institute may develop a traditional institute option where teams of faculty from one or more institutions, or consortia of entities, can come together and learn how to engage in scoring, rubric use and representation of results. In the meantime, the VALUE Institute will be inviting participant organizations to recommend interested individuals for training to become certified VALUE scorers. Also, the VALUE Institute will offer to conduct scorer training to build local capacity for groups of interested individuals to the extent possible as the Institute begins formal operations.
PRICING
How much will it cost to participate in the VALUE Institute?
The work of VALUE has evolved from standalone rubric tools to demonstrating the proof-of-concept through the three Collaboratives, and is now poised to transition to a self-sustaining Institute. Cost for assessment of student work artifacts through the VALUE Institute is $6,000 to assess 100 pieces of authentic student work. This fee covers the expenses to:
- Assist with developing a sampling framework and strategy
- Upload student work artifacts in the Institute database (Watermark/Tk20/LiveText platform – you do not currently have to use these platforms)
- Address one learning outcome (VALUE rubric) across each rubric dimension for 100 pieces of student work
- Have all artifacts double-scored by certified VALUE rubric scorers
- Generate scoring results reports, report templates, as well as the raw institutional data from scoring
The fee for each additional outcome (and 100 artifacts) is $4,000
Why does the VALUE Institute cost $6,000?
There are two primary drivers behind the cost of the VALUE Institute: (1) access to Aqua, the technology platform developed to enable the uploading, distribution, and scoring of student work, and (2) the actual scoring of the student work with a VALUE rubric. AAC&U firmly asserts that human scoring of the student work is a critical piece of the VALUE Institute. Not only is it a key component of the VALUE approach to assessment from a methodological perspective, scoring student work also serves as a unique faculty development opportunity. Scorer training and the scoring of actual student work also doubles as faculty development that addresses such things as assignment design to pedagogical strategies in the classroom. VALUE scorers are remunerated for their work with the VALUE Institute. For more information on scoring, please see the section on FAQS on Scoring.
Does the VALUE Institute offer discounts based on characteristics like institution size?
The VALUE Institute is predicated on a set number of submissions from all institutions/providers, regardless of size. It is the number of submissions that drives the cost as it determines the number of scorers needed to do the actual scoring of student work.
Is there an annual membership fee?
When do we have to pay?
Payment is due when an institution has successfully uploaded its 100 artifacts for scoring, which must be completed by May 1, 2018.
PARTICIPATION
Who is the VALUE Institute designed to serve? What types of institutions should participate?
The VALUE Institute serves the learning outcomes assessment needs of institutions, departments, membership organizations, research projects and other higher education initiatives. Participation in the VALUE Institute is open to all types of higher education providers, consortia, institutions and departments within institutions.
What proportion of participating institutions do you expect to be from community colleges?
In initial development of the VALUE Institute, community colleges have been about half of the overall participants in the grant-funded, proof-of-concept phase of VALUE (2014-2017; see the section on Additional Questions for history of the VALUE Institute).
Is participation limited to AAC&U member institutions?
No. AAC&U membership is not a requirement for participation in the VALUE Institute.
Can we participate in the VALUE Institute as a consortium?
Any individual campus or program within a consortium may participate, and campuses may indeed choose to participate as a consortium in the VALUE Institute for purposes of data sharing, resource sharing, etc. Additionally, there may be opportunities to engage with such groups as the Multi-State Collaborative (MSC), an on-going collaborative partnership between AAC&U and SHEEO, for two- and four-year public institutions. Institute staff are willing to assist institutions in thinking through possible consortia arrangements.
Could we form a consortium within the national norms for specialized disciplines (e.g., visual and performing arts)? Can individual departments or groups of faculty participate independently of the broader institution?
Yes, the VALUE Institute is open to participation from disciplinary/departmental/program entities, states or consortia.
Can institutions outside of the United States participate?
Yes, the VALUE Institute is open to participation from non-U.S. entities.
How does an institution join the VALUE Institute?
How does the VALUE Institute work?
For each learning outcome selected, a total of 100 artifacts can be uploaded. Participants can work with Institute staff to determine the best way to identify and collect student work samples/artifacts that address the specified learning outcome. Once student work is collected, demographic information about the students and some information about the assignment/course/source of the work is also collected. These data are combined into a single file for batch upload into the Institute system platform for scoring and data analysis reporting. Collection of work may be from any given semester/term. Each artifact is double scored by scorers. The collection of student work for uploading through the VALUE Institute involves demographic information associated with the students who produced the selected artifacts. This process allows both the Institute, as well as the local entity, to do analyses and disaggregation of the results. The results are provided back to the submitting participant in a database that can in many instances be connected to local SIS or LMS systems. Our platform – Watermark/Tk20/LiveText – can communicate with the systems many entities are already using.
How labor intensive is participation?
There is work involved by participants. Sampling plans need to be developed (VALUE Institute staff will help with this as needed) to ensure work samples will yield results that appropriately answer questions the participant wishes to address. Additionally, demographic information will need to be collected based upon the sampling plan for students whose work is collected, course information, faculty assignments that prompted the work, and faculty intent regarding the assignment and the learning outcome.
In short, VALUE certified scorers will score the uploaded student work. The work artifacts can be scored locally (we encourage this) but it is not required. The participating organization samples, collects, and uploads the student work.
What resources are available to help an institution effectively participate?
The VALUE Institute would not be possible without the contributions made by the three VALUE Collaborations that pioneered this approach to assessing student learning. The MSC provided intellectual and pedagogical leadership to this approach, which is reflected in the suite of resources available to campuses as they onboard with the VALUE Institute. Specifically, campuses will be provided with sampling frameworks, templates for submitting required data, and assignment "cover sheets" when working with faculty to identify and select appropriate work samples for submission to the Institute. Webinars and asynchronous video resources will be provided as well. Finally, the VALUE Institute has identified a cadre of campus-based VALUE Institute Fellows who can be contacted for advice and counsel regarding their experiences implementing the VALUE approach on their campuses.
Which office/department should be involved in the VALUE Institute? Does it need to be the institutional research office, or can it be a teaching and learning center/scholarship of teaching and learning institute?
It is up to each institution to determine how to distribute the work and who is involved. Offices that have commonly been involved in some aspect of this work in the past include assessment offices, offices of institutional research, the registrar, centers for teaching and learning, and/or program directors/department heads/academic deans (e.g., the director of the writing center, the associate dean for undergraduate education, etc.)
What is the timeline for uploading artifacts and receiving data back?
Institute scoring initially occurs once per year in the summer. Therefore, uploading artifacts must be completed by May 1 or 15 (late spring unless pre-approved extension is approved). The VALUE Institute generated reports as well as an institution's/provider's raw data will be available at the beginning of the fall term.
METHODOLOGY AND SCORING
What is "standardized" in the VALUE Institute? What is not?
Since the work of the Institute is designed to reveal a landscape of learning across higher education and essential learning outcomes that can be disaggregated by institutional type, state/region, gender, etc., all student work must be scored using a common set of standards. In this case, the common standards are the VALUE rubrics. By using a common set of measures, reliability and validity associated with the landscape of learning is enhanced, and thus provides higher confidence levels for the data.
There is no common prompt or required assignment template for the student work that is submitted for scoring in the VALUE Institute. AAC&U believes firmly that the most valid approach to assessing student learning requires us to honor faculty expertise and raise up assignments that come directly from the classroom or co-curricular experience.
How is a sample of 100 pieces of student work "representative" of learning on a campus?
It is likely not, depending on the size of the institution. The VALUE Institute is not designed to allow a campus to "outsource" its assessment of student learning. Rather, the VALUE Institute provides the unique opportunity for institutions/providers to engage in the external validation of locally-conducted assessments. Participants are strongly encouraged to have their own trained educators score locally the same sample of student artifacts submitted through the Institute, using either their own adapted rubrics or the original VALUE rubrics to reveal differences that may be due to rubric changes, while also highlighting similarities. Even using adapted, local versions of learning outcomes rubrics can be viewed and compared to nationwide benchmarks emerging from the VALUE Institute process.
For the external validation of the demonstration of learning, the original VALUE rubrics will be used through the Institute. Many institutions have used this approach to establish a baseline of learning as they have experimented with modifying VALUE rubrics for their own mission, or developing variations of them to be able to explore how rubric changes influence student demonstrated learning. In addition, faculty have found it useful to review and tweak assignments to strengthen areas of learning outcomes to encourage demonstration of student learning.
Who are the scorers?
The VALUE Institute scorers are recruited from institutions who have participated in VALUE scoring in the past or are participating in the Institute. Scorers are recruited from campuses/entities so that they represent a diverse set of individuals, institutional types, disciplines, etc.
They are trained by AAC&U trainers using the established AAC&U protocols for scoring, including calibration/norming training in the use of the VALUE rubrics. Scorers' scores have been examined to confirm calibration on the VALUE rubrics prior to certification to ensure standards of reliability and validity for the results.
Because we've been scoring for many years, we have a number of people that have experience with the VALUE Rubrics. How does one become a "certified" VALUE Institute faculty scorer?
Any participating entity in the VALUE Institute will be invited to recommend their colleagues to become trained VALUE scorers with the possibility of becoming a certified VALUE scorer.
Are artifacts double scored? What is the inter-reader agreement (or even reliability) of the normed readers?
Yes, all artifacts are double scored. Scorers do not know where the artifacts they score originate. AAC&U investigated the reliability of the VALUE assessment process with several small-scale studies between 2008 and 2010. Through these processes, slightly different patterns of scoring emerged depending upon the disciplinary differences of the scorers. That said, there were no statistically significant differences across the group—demonstrating that faculty from a range of disciplines could indeed score student work from within or beyond their own discipline and reach relatively high levels of agreement.
As part of the Demonstration Year (2015-2016) for the VALUE initiative, inter-rater reliability was an important methodological concern. Approximately 20 percent of the work samples submitted for Written Communication and Critical Thinking were double scored, with nearly all the Quantitative Literacy work samples double scored. While there are a range of statistical tests available to ascertain inter-rater reliability, preliminary examinations included weighted percent agreement between raters, weighted Cohen's Kappa (which, for a design like the VALUE approach to assessing student learning, is limited in many ways), and Gwet's AC, which adjusts for chance by considering how difficult or easy it is rate a subject. Using Gwet's AC, the range of agreement across the dimensions for Critical Thinking, Written Communication, and Quantitative Literacy shows that agreement can be described as moderate to strong.
Please see pages 26-32 of AAC&U's report,On Solid Ground, for more detail.
What types of student artifacts can be scored?
There are some general parameters on assignments (available through the Institute web site) and the resultant work that can be uploaded, e.g. currently artifacts must be in English, page lengths may apply, number of artifacts from a single class or instructor, etc. The Institute will work with participants to go over the parameters related to assignments and the rubrics. Within the parameters, it is the intent of the Institute to have participants collect existing work from existing assignments to be uploaded and scored so that no additional time and effort is expended by faculty or students in generating the artifacts. That being said, the parameters do provide guidance on appropriate assignments and artifacts that facilitate usefulness for the scoring results, their utility for change and improvement for learning.
REPORTING AND DATA USE
What level of reporting will institutions receive and when?
Reports will contain detailed information for overall rubric scores and each rubric dimension. Reports will also disaggregate results by student demographics and provide summary results for relevant groups of providers (e.g., other participating providers in a state). Reports will be delivered in late summer or early fall.
What will the VALUE Institute do with all the data it collects?
Individual institution/entity results are only shared with the originating entity and whoever they designate as appropriate to receive the results. By participating, participants agree that all anonymized scoring results and data can become part of the nationwide benchmarking database. No specific institutional identifiers are maintained with the results and data. Anonymity and confidentiality on the part of the Institute are part of the agreement to participate. Rankings are not part of the Institute's reporting or representation of the results. Individual student and participant scores will be kept in confidence and only shared with the participant. Only aggregate or summary scores from across groups of institutions will be shared.
How is student data protected?
The VALUE Institute does not collect any identifiable information on students or faculty. Campuses are asked not to submit actual student names or ID numbers. Many campuses choose to create a code to represent each student whose work is submitted, so that the data can then be connected back to campus student information systems, locally conducted assessment results, etc. Additionally, the VALUE Institute does not collect faculty names or identification numbers. We do, however, collect data Demographic data to allow for disaggregation to address patterns of learning and to address issues of equity, e.g. gender, race, Pell eligibility, credit hours earned, etc. We also collect course information such as level and subject/disciplinary area, the assignments that generated the student work, as well as assignment-level information regarding faculty intentionality vis-Ã -vis the criteria of the VALUE rubric against which student work samples will be scored. These data are critical to our collective ability to contextualize the results generated through the VALUE Institute broadly, as well as on individual campuses specifically.
The assignment itself is never provided to scorers or other participants. Each individual piece of student work is only shared with the certified VALUE scorer(s) assigned to score it, and scorers ONLY see the work sample itself, not the associated assignment or any other data. All scorers must sign a confidentiality agreement in order to have access to the scoring system.
What types of reports will be generated and made available to campuses?
Reports will allow participants to contextualize their findings within the higher education landscape and benchmark against the rubric criteria. Benchmarking against particular sub-groups or types of providers will depend on participation. Participants receive back their own raw data as well as a set of scoring results and templates showing aggregate results on all learning outcome dimensions, as well as for disaggregated participant categories for student demographics where appropriate.
Will assessment results be disaggregated by performance criteria (indicators) for each rubric?
Results will be disaggregated by rubric dimensions and student characteristics, where possible.
Will I be able to use the assessment information towards funding opportunities or in published papers?
Participants will be able to use their data and results for whatever purposes they deem appropriate.
Can we get feedback and suggestions based on results?
Yes, the VALUE Institute plans to provide templates, visualization suggestions and feedback on the meaning and framing of the results along the lines of the NSSE Annual Reports and Lessons from the Field provided to NSSE institutions. The Institute will encourage follow-up and mentoring (provider to provider) and, when capacity allows, provide its own consultations, follow-up, and mentoring of providers.
Will participants have access to other institutions' best practices? Will the Institute share findings related to learning outcomes?
The VALUE Institute will develop and host webinars on best practices among participants as part of Institute participation and share write-ups of best practices through its website and reporting.
Will training be available on how to make the best use of data analyses?
Yes. Each year there will be conference workshops and, potentially, summer institutes aimed at conducting and sharing analyses as well as understanding and using results.
We are looking towards moving to embedded assessment for general education. Would this program help validate our internal assessments?
ADDITIONAL QUESTIONS
How did the Institute come about?
Since 2014, the VALUE initiative has moved beyond local, individual campus implementation to also test its applicability as a framework for intra- and interinstitutional, inter-sector (e.g., two-year, four-year public, four-year private), and interstate assessments of student learning. The foundation of the VALUE Institute is the Multi-State Collaborative (MSC), which was the centerpiece of the VALUE work, and currently involves 13 states. The MSC, in turn, is based on the "Vision Project," which started in Massachusetts. There, a broadly collaborative, multi-campus leadership group worked to conceptualize a model for state system assessment based on the LEAP Essential Learning Outcomes and using VALUE rubrics. Massachusetts leaders wanted to see if other states would join the effort to assess learning outcomes and voluntarily share results, so they reached out to State Higher Education Executive Officers (SHEEO) to help recruit other states into the MSC work. These leaders recognized that state (and national) higher education policy should be informed by and rest upon solid evidence of the quality of student learning in postsecondary education. AAC&U, SHEEO, and the participating states and institutions of the MSC believe that the public, policy leaders, legislators, and employers deserve to know whether students receiving credentials from higher education institutions can demonstrate the expected and necessary levels of proficiency on a full range of learning outcomes central to life and workplace success.
To date, through three related consortia— the MSC, the ten-institution Minnesota Collaborative, and the nine private four-year colleges in the Great Lakes Colleges Association (GLCA) Collaborative—ninety-two institutions have submitted 21,189 student work products (representing 3,051 unique faculty-designed assignments) for assessment by 288 faculty trained to score the student work using one of six VALUE rubrics, with the most popular outcomes rubrics being Critical Thinking, Quantitative Literacy, and Written Communication. VALUE provides information that can inform decisions by local, state, and federal policy makers for improvement. Simultaneously, the findings point to actions that can be taken by those directly involved in teaching and learning on a day-to-day basis—faculty, other educators, and students—to effectively focus attention on achieving even better results. This work was financially supported by funders such as the Gates Foundation, the Sherman Fairchild Foundation, the Spencer Foundation, and the Lumina Foundation.
Is the VALUE Institute duplicating work conducted by Achieving the Dream (ATD)?
No, it does not. The VALUE Institute supports using data to inform decisions and practices for student success; however, the Institute fills a gap in the data by providing valid and reliable information on the learning students demonstrate on essential outcomes regardless of the type of institution, program of study, or mode of learning. Results can also be disaggregated to reveal patterns of equity in demonstrated learning for specified populations of students.
Does the Multi-State Collaborative (MSC) still exist? What is the relationship between the VALUE Institute and the MSC?
It is the hope and expectation that the MSC not only continue to exist as a collaboration that utilizes of the VALUE Institute, but that it continues to be the centerpiece of this work, as it is uniquely positioned to address questions at the nexus of student learning, quality, and completion agendas at the state policy level. States are going to be strongly encouraged to have a representative sample of institutions in their state participate to provide generalizable results for state policy making. Questions regarding participating in the VALUE Institute as part of the MSC should be directed jointly to Kate Drezek McConnell, Senior Director for Research and Assessment at AAC&U (mcconnell@aacu.org) and Denise Pearson, Principal Policy Analyst, State Higher Education Executive Officers Association (SHEEO) (dpearson@sheeo.org).