A credentialing process for advanced level pharmacists: participant feedback

Introduction

As drivers for health services changes continue to grow, the knowledge and skills required to deliver those services will inevitably change.^[1]
 Although studies on this matter are not conclusive, the implementation of a credentialing (or professional recognition) process for advanced expertise may help both practitioners and healthcare employers to ensure better patient safety, better quality of care and better outcomes.^[2]
 The concept of credentialing is already in use in some healthcare professions and can be defined as “a quality assured process that recognises a practitioner’s attainment of the required knowledge and skills at a particular level of practice. Crucially, this is a process conducted through professional peer review, and is not connected with a regulatory function. It exists for the purposes of validation of practice by peers, and demonstrates a recognition of practice which has value and merit for the general public and other members of the profession or professional colleagues.”^[2],[3]
 Furthermore, professional recognition and credentialing might be applied for three different purposes:

• To support practitioner development and progression through training
• To support revalidation of practice
• To ensure a quality assured, robust evaluation of advanced levels of practice^[2]
  
  
  

The United Kingdom Clinical Pharmacy Association Critical Care Group (CCG) was asked by the association to create and test a multi-source tool to assess, credential and recognise expertise in advanced practice and to evaluate the outcomes of such process.^[3]
The Advanced and Consultant Level Framework (ACLF) and the Critical Care Curriculum Framework (CCCF) were the foundations of the methodology developed.^[4],[5],[6]
 In order to test this process, 17 critical care pharmacists volunteered to participate in the prototype credentialing method. A group of examiners, comprising consultant pharmacists in critical care, senior academics and senior NHS service managers, was selected by the project team. Candidates were assessed via a set of tools, which included a structured practice portfolio, case-based discussions, clinical evaluation exercise and 360-degree peer review, and were subject to a final evaluation of their level of practice against each of the six ACLF main cluster domains.^[3]
 Moreover, it was proposed that this core process could be applicable for use in wider circles of the pharmacy profession, including general level practitioners.^[2],[3] The full process review and conclusions on the method has been described and published previously.^[3]

Further work needs to be undertaken regarding credentialing efficacy and correlation with clinical outcomes.^[2]
 So it is of interest to assess the candidates’ feedback on the overall experience and the impact that the process had on the continued development of their practice and careers. Candidates’ opinions and perceptions of the process will be important for future adjustments, uptake and implementation in other specialties.

The aim of the study was to understand the impact of this professional recognition model in pharmacy practice by assessing the impact and effect on candidates’ professional roles and career perspectives.

This feedback audit was intended to assess the candidates’ experiences and perceived impact of the credentialing process through a qualitative survey. The practitioner candidates who experienced the prototype credentialing model were asked to describe the effect of the process on their practice and career perspectives, in terms of recognition of knowledge and skills by other professional and non-professional colleagues, increased professional flexibility within health institutions and quality of care delivered. Candidates had the opportunity to report difficulties they may have perceived and make suggestions. The main objective was to gather information in order to clarify some aspects of the method that may need improvement. Also, candidates were asked about the changes they have made in their practice after the evaluation process. Candidates were drawn from the three cohorts who had volunteered for inclusion in the UKCPA CCG project in 2009, 2010 and 2011.

Method

Sample

The audit sample consisted of all the critical care pharmacists (n=17) who had been assessed via the credentialing process. One of the candidates went through the process more than once (2009 and 2010) to check for consistency, however, for this audit was considered as a single candidate. They were invited to participate via an email comprising an introductory message, information on the intent of the survey, how it would be conducted and assurance that confidentiality would be ensured.

Data collection

Data were collected through a structured and selective process, in the form of an online survey.^[7],[8]
 The survey included closed questions to elicit demographic information and open questions to allow respondents to describe their relevant experiences and to provide a basis for an in-depth analysis. The survey questions are given in Panel 1.

Data analysis

In order to analyse the demographic data, the ANOVA (with Bonferroni) test was performed, using the SPSS version 17 (NYC). The qualitative data were analysed using a “matrix” qualitative approach (Miles and Huberman^[7]
), thus the information was assessed using an iterative coding and tabulation process. The iterative process allowed for the identification of principal themes within the data and associated thematic codes, which were grouped into three main themes to form a coding frame (Figure 1).

All responses were tracked using the data reference ID and were mapped to the cohort, in order to provide context for analysis. The tracked and coded data were displayed in a matrix format containing the themes, thematic codes and the respective coded segments of representative data. This first iterative matrix was then subject to a sorting process, within the matrix, in which responses were categorised into positive or negative commentary within each theme (“matrix by theme”). This process was validated by a second researcher before continuation. A further version of the matrix was then created by subdividing positive and negative columns into the relevant candidate cohort (“matrix by cohort”) where a cohort was defined as the year in which candidate was subject to the credentialing process (2009, 2010 or 2011). The sorted data from the “matrix by theme” was subjected to a weighting process, which was based on strength and depth of positive/negative coded text within the data and the relative quantity within the coded excerpts. This developed into the final version of the matrix. All steps of the iterative and weighting process were verified by a second researcher for credibility.^[7],[8]

Conclusions were drawn from the coded excerpts and weightings, and were confirmed, tested and verified against the raw data by two researchers, in order to ensure continued credibility and reliability of the analytic process.

Results

Seventeen pharmacists (nine in grade 8A and eight in grade 8B) took part, five in the 2009 cohort, five in 2010 and seven in 2011.

Within the three cohorts, candidates in the first cohort had more experience in pharmacy practice (p=0.001) and also in critical care environments (p=0.005). There was little difference in these aspects between the two subsequent cohorts in 2010 and 2011 (Table 1).

The themes and thematic codes extracted from the raw data through an iterative process are represented in Panel 2. The two matrices, “Matrix by theme” and “Matrix by cohort”, are too large to reproduce here. Copies are available from the authors on request.

Thematic descriptions

After conducting the iterative coding process, three principal themes emerged from the data; aspects of the evaluation and credentialing process (process context); the practitioners’ own context (personal expectations); and the practice context for the practitioner (professional context).

Process context 

The candidates generally considered the assessment environment to be appropriate, although with some limitations, such as limited space and a “busy” clinical department. Nonetheless, the process used the environment appropriately. There were some concerns about the lack of familiarity with the department, staff and electronic system (for the visiting candidates) which may have affected candidates’ performance on the assessment day. Consideration should be given to using paper notes or provide a local “helper” for using any electronic record systems (which tend to vary between locations):

Duration was perfect; there was a lot to fit in and spreading it over a day made it less intense. Doing it in a working environment made it more real to life and so simulated the pressures of a “normal day”… . A home environment would be impossible to organise so I think if the process was done again a similar set up would be the only feasible option. (2010 cohort)

It was stated there was a good balance between time and thoroughness:

Tiring day, but enough time to fit everything in. Environment was fine, although potentially quite hard to get used to computerised system, despite someone navigating for you. (2010 cohort)

The credentialing model was seen as a valid and robust process, as it evaluated candidates in a working environment, with different types of assessments and different assessors, which reduced bias:

I thought it was well organised and the assessment types gave an opportunity for assessment in different environments with different assessors and different modes of assessments all of which serve to assess the whole practitioner. (2011 cohort)

More guidance and support for portfolio construction is needed. It was seen as a time consuming task (perhaps due to this being the first time a practitioner had been required to construct a practice portfolio) and possibly a good idea to create a standard template, in order to facilitate construction and review:

Developing a standard template electronic portfolio would be of benefit to both assessors and attendees, especially those developing portfolios from scratch for the purposes of this process. Time taken to review portfolios would be significantly reduced by standardisation. (2010 cohort)

It was seen as useful to be assessed by peers, who are respected across the profession. However, “examiners” need training and development to ensure quality in the process. A more relevant issue for the first cohort (year 2009) perhaps due to the “novelty” of the examiners and these comments were not seen in subsequent cohort assessments, which reveals improvements in the process:

Mine was the first version and it was fairly chaotic as it was all new to the examiners. I felt that the viva was unnecessarily confrontational and aggressive and not very skilled in asking the right questions. On subsequent sessions we have had a moderator to at least talk through before hand what was reasonable to expect. (2009 cohort)

It is really useful to be assessed by your peers and people whose opinion you respect. However I think it is important to be precise about what is expected from the candidate, eg, the viva. I was also expecting questions rather than for me to justify my level of expertise. (2010 cohort)

The process is useful to understand what is expected from pharmacists at advanced levels of practice. However, it needs to be made more clear what is expected from candidates, and it is important for the process to have clear structure and consistency:

It was a good attempt at finding out what is expected of pharmacists in this area from specialists i.e. pharmacists and clinicians. (2010 cohort)

Personal expectations 

The outcome of the process tended to meet most personal expectations. Most also considered the feedback to be useful, reflective, practical and constructive:

Yes, but I would say the evaluation outcome was more positive than I expected. (2011 cohort)

The process is good for identifying areas for improvement and ways of approaching and developing them. It was seen useful as guidance for training and improvement purposes.

It gave me reinforcement of the areas I need to focus on to better round my practice, which were in line with what I had thought, but was reassuring to have someone else reinforce this. Research was one of the areas I knew I needed more work in and this was reiterated by the assessment — hence my priority for the next six to 12 months is to take the many projects I have on the go to publication. (2011 cohort)

The credentialing process was seen as an incentive for reflection, both for the individual candidates and for local managements and organisations. The process led to changes in practice by means of action on feedback or subsequent reflection and, this way, improvement in practice:

The structure of the advanced practice guidance and building of the portfolio were useful to help me think where I should develop my role in the future. (2009 cohort)

I was advised to progress my research activities to attempt to get this funded … subsequently I am a grant holder on a NIH grant of £220,000 study, so I have made progress in this field. (2009 cohort)

The credentialing process leads to longer term reflections and subsequent actions. In addition, the outcomes should lead to more sustainable and effective changes in the practice of pharmacists and in organisations.

Professional expectations 

Managers and peers were impressed by the method and considered it useful and valuable, and created conditions for consideration in appraisals and advancement in career. Also, the process increased the candidates’ confidence in their own knowledge and enabled the demonstration of level of practice in a formal manner.

I discussed it with my manager at my IPR review. They were impressed by it and thought it was a good idea and a good thing to have been involved with. (2010 cohort)

However, the most important impact is on my own understanding of the level of my practice and being able to demonstrate this in some concrete form to my managers who have varying degrees of understanding of critical care practice. (2011 cohort)

The importance of the method was also revealed in identifying the level of practice and positioning within the ACLF. The construction of a practice-based portfolio together with mapping to the ACLF was seen as very useful, as well as the examiners’ feedback, as a way of validation of self-assessment:

Preparing a portfolio and matrixing it to the ACLF gives you a realistic idea of where you are at before you attend. The results I got back were pretty much as I expected … with a good idea of where you are at. (2010 cohort)

The process was useful in identifying and prioritising areas for improvement. Guidance is needed, for education and training purposes, for pharmacists in advanced levels of practice:

I think it has shown that some kind of formal training should be incorporated into pharmacists in permanent posts. I say this because not every hospital [has] established ICU pharmacists who can train a junior who could then go on to take a senior post in another ICU. (2010 cohort)

The credentialing process was seen as useful and relevant to career progression. It was helpful for prioritisation of areas for improvement, and it incentivised career progression. However, there is a need for creating conditions to implement the process equally across the UK.

The structure of the advanced practice guidance and building of the portfolio were useful to help me think where I should develop my role in the future. (2009 cohort)

The credentialing process has had a large impact on my approach to professional development and would almost certainly improve the career prospects of those who attend. (2010 cohort)

Profound as it identified where I am now and what I need to achieve in the future to help me model my future career. (2011 cohort)

The process led to improvement in career perspectives and was considered a better approach to career development, and acted as a drive for a change in institutions:

I hope it will encourage my department to facilitate some development in the future towards a consultant post … however I will be surprised if they actually do. (2011 cohort)

In order to summarise all the data and enable a visual demonstration of the results, a weighting approach was used to create a “density chart”. This chart represents the three main themes extracted from the data, which were then mapped against the respondents cohort and the relative weighting of the responses extracted from the coding matrix. A summation of all the positive/negative responses within the matrix provided a final weight which was charted as in Figure 2. Darker cells correspond to more positive responses whereas lighter cells correspond to less positive or negative responses.

Discussion

The “process context” of the credentialing event was the least positive set of experiences drawn from this respondent sample. It was rated lowest by candidates from the first cohort, probably due to inexperience of the assessors of this first cohort, and was still the least positive theme in later cohorts. Respondents mainly referred to a lack of familiarity with the environment, issues with the portfolio construction and stresses during the assessment day. The 2009 cohort was a pilot and had some particular issues, as it was the first time the process had been used in practice. Examiners were not fully prepared for the demands of their task and in retrospect needed training and more preparation. However, these candidates were the most experienced in their field, and may not be representative of more typical types of pharmacists.

More positive responses emerged over with the subsequent cohorts, showing that the CCG team was learning and gaining experience as each iteration was developed, building up real-time experiences of managing the process. Overall, most candidates considered the process as providing a valid and robust assessment of the practitioner, with minimal examiner bias.

Professional expectations emerged as being positive, increasingly so from 2009 towards 2011. Candidates from all cohorts agreed that the process is useful to define their level of practice and positioning within the ACLF, thus increasing confidence in their own knowledge and validation of self-assessment. Moreover, having undergone this process, candidates realised the need for more formal education, training and guidance for pharmacists in advanced levels of practice, as these pharmacists may not have had access to peer review and formal learning in their field. In this way practitioners noted increasing confidence in providing evidence of level of knowledge to managers. The credentialing process was seen as relevant and developmental for career progression, as it helped to focus and improve practice.

Personal expectations showed a similar trend and increased in positivity over time. Opinions on the expected outcome are relatively more positive in 2011 cohort than in 2009 cohort. However, candidates from all cohorts agree that the process was a good support for identification and management of areas for improvement. Written feedback and follow-up guidance were offered to the first cohort, but not every candidate took up this offer; however, more formal feedback provided for all candidates in the later cohorts, who considered it to be extremely useful. Furthermore, the process worked as an “incentive” to both individual and collective reflection on career pathway and perspectives, leading to real changes in practice. Undoubtedly, the process entailed longer-term issues, as reflections on outcomes and subsequent actions do take some time to develop into a sustainable and effective outcome for both pharmacists and organisations.

Finally, professional and personal expectations were most strongly mapped to the most recent cohort (year 2011), implying that the process, as a whole, was evolving to meet the aims and objectives of the process as a validation tool for practice in this stratum of practitioners.

These results provide confidence that a credentialing process is both feasible and useful for clinical practitioners, and provide a measure of competent performance for pharmacists while the pharmacy profession in the UK evolves. There is more to be learnt about the management of a credentialing process, but the evidence so far indicates that the UKCPA CCG is gaining valuable experience in this field of practitioner development.