Usability of an internet-based platform (Next.Step) for adolescent weight management

Abstract

Objective

The current study evaluates the usability perception of an e-therapeutic platform (supported by electronic processes and communication), aiming to promote the behavior change and to improve the adolescent health status through increased and interactive contact between the adolescent and the clinical staff.

Methods

This was a correlational study with a sample of 48 adolescents (12-18 years) who attended a Pediatric Obesity Clinic between January and August of 2012. Participants were invited to access, during 24 weeks, the e-therapeutic multidisciplinary platform (Next.Step) in addition to the standard treatment program. A usability questionnaire was administered and the platform performance and utilization indicators were analyzed.

Results

The users’ perception of satisfaction, efficiency, and effectiveness regarding the Next.Step platform was clearly positive. However, only 54.17% of the enrolled adolescents accessed the platform, with a mean task-completion rate of 14.55% (SD = 18.853). The higher the number of the platform consulted resources, the greater the tendency to enjoy the platform, to consider it exciting and quick, to consider that the time spent in it was useful, to consider the access to information easy, and to login easier. Post-intervention assessment revealed a significant reduction in anthropometric and behavioral variables, including body mass index z-score, waist circumference percentile, hip circumference, and weekly screen time.

Conclusion

These results highlight the importance of information and communication technologies in the health information access and the healthcare provision. Despite the limited adherence rate, platform users expressed a positive overall perception of its usability and presented a positive anthropometric and behavioral progress.

Resumo

Objetivo

O estudo atual avalia a percepção de usabilidade de uma plataforma e-terapêutica (suportada por processos e comunicação eletrônicos), visando promover a mudança comportamental e melhorar o estado de saúde dos adolescentes através do contato aumentado e interativo entre o adolescente e a equipe clínica.

Métodos

Estudo correlacional com uma amostra de 48 adolescentes (12-18 anos) que frequentaram uma Clínica de Obesidade Infantil entre janeiro e agosto de 2012. Os participantes foram convidados a aceder, durante 24 semanas, à plataforma multidisciplinar e-terapêutica (Next.Step) e, além disso, a participar do programa de tratamento padrão. Um questionário de usabilidade foi administrado e foram analisados os indicadores de desempenho e utilização da plataforma.

Resultados

A percepção de satisfação, eficiência e eficácia dos usuários com relação à plataforma Next.Step foi claramente positiva. Contudo, apenas 54,17% dos adolescentes inscritos acederam à plataforma, com uma taxa média de conclusão das tarefas de 14,55% (DP = 18,853). Quando maior o número de recursos consultados na plataforma, maior a tendência de gostar da plataforma, considerando-a excitante e rápida, considerando que o tempo gasto na mesma é útil e considerando o acesso a informações e ao registro mais fácil. A avaliação pós-intervenção mostrou uma redução significativa nas variáveis antropométricas e comportamentais, inclusive no z-score do Índice de Massa Corporal, no percentil de circunferência da cintura, na circunferência do quadril e no tempo de tela semanal.

Conclusão

Esses resultados destacam a importância das Tecnologias de Informação e Comunicação no acesso a informações sobre saúde e prestação de cuidados. Apesar da limitada taxa de adesão os usuários da plataforma expressaram uma percepção geral positiva de sua usabilidade e apresentaram um progresso antropométrico e comportamental positivo.

Introduction

Healthcare is experiencing a cultural transformation, challenging physicians to change from a provider to a patient-centered healthcare model perspective. This switch is made possible by the evolution of information systems in the field of health sciences and empowered by the e-health approach (healthcare practice supported by electronic processes and communication), which contributes to a more efficient, accessible, and affordable healthcare model.¹ Therefore, internet plays an important role in this healthcare paradigm change, with patients becoming empowered e-consumers (consumers of electronic processes and communication).

There has been a growing use of technologies regarding the search for health information.^2,3 One of the benefits of e-health interventions is its flexibility, which may facilitate the provision of tailored messages based on the users’ specific cultural and individual contexts. The interactive nature of e-health applications can further enhance learning and behavioral change.⁴

A systematic review on the effectiveness of information and communication technology (ICT) in the treatment of obese children and adolescents in the United States reported that 75% of the online intervention programs yielded clinical and statistical significant results. These results underscore the effectiveness evidence in reducing body mass index (BMI) and fat intake and in increasing physical activity.⁵

Interactive technologies offer new opportunities to provide tailored health information. However, usability should be the main goal of an interactive product, so that it can be employed for its intended purpose. Increased usability brings significant benefits: increased efficiency, improved productivity, reduced errors, less need for training, and better acceptance.⁶ Considering usability as a dynamic process, its assessment is dependent on how it is defined.

There are different approaches regarding to the usability indicators; Hix & Hartson,⁷ whose vision was adopted by the International Organization for Standardization (ISO) 9241-11,⁸ defines usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness (meaning both the ability of completing tasks through the system and the quality of the achieved results), efficiency (level of resources consumed in performing tasks), and satisfaction (subjective reactions of users based on the use of the system) in a specified context of use.

There are still relatively few contributions on the usability of e-health systems specifically designed to promote healthy behaviors and weight reduction. One of them is the study of Shaikh et al.,⁹ which evaluated the usability of three websites dedicated to weight loss, making a comparative analysis of its features and performance. Tsai et al.¹⁰ also studied the usability of mobile phones as tools for monitoring nutrition and exercise in real time. Compared to paper recording, they found that mobile users reported equal or higher usability rates.

The current study evaluated the usability (satisfaction, efficiency, and effectiveness perceptions) of an e-therapeutic platform (Next.Step), which aims to promote the behavior change and to improve the adolescent health status through increased and interactive contact between the adolescent and the clinical staff.

Methods

Sample

Participants were adolescents followed at a hospital-based Pediatric Obesity Clinic, fulfilling the following inclusion criteria: aged between 12 and 18 years, BMI percentile ≥ 85th, and with internet access at least once a week. Sample recruitment had the support of the clinical staff. All eligible adolescents with appointments between January and August of 2012 were invited to participate in the program.

Participants followed the standard treatment protocol and, additionally, received free access to the e-therapeutic platform (Next.Step) for 24 weeks. They were divided into two subgroups, depending on the study enrolment order: group A (n = 23) and group B (n = 25). Group A began the intervention in May 2012 and Group B in October 2012, aiming to reduce the adolescents’ waiting period to start the program. However, the procedure was identical in both groups, namely the treatment length, methods, content/resources, and functionalities. The case manager and the clinical team were the same in both groups.

Procedures

This study was approved by the Ethical Committee for Health (Lisbon, Portugal) and funded by the Foundation for Science and Technology (PTDC/DTP-PIC/0769/2012). All eligible adolescents and respective parents signed an informed consent where the study objectives were explained, following the ethical aspects of confidentiality and voluntary participation.

Data analysis

Descriptive and inferential statistics were applied (paired and independent tests), considering the study aims and variables. Nonparametric tests (Spearman's correlation, Mann-Whitney's U test, and Kruskal-Wallis test) were used in inferential statistics, due to the reduced number of valid responses in several indicators. All analyses were conducted using bilateral tests and alphas of 0.05. For this purpose, was used the software SPSS, version 22.0 (IBM Corp, NY, USA).

Measures

Survey methods and platform functioning have been previously described in detail elsewhere,¹¹ as this study is part of an ongoing longitudinal research. Data were collected through different sources, namely: a) the clinical file (demographics, anthropometric, behavioral, and clinical variables); b) the Next.Step e-therapeutic platform (utilization rate, number of logins, consulted resources, self-monitoring data, game scores, and participation in forums); c) a self-report instrument (Next.Step Usability Questionnaire [NSUQ]).

BMI assessment. Anthropometric data were measured by trained health professionals from the clinic. Adolescents were classified into overweight and obese according to BMI cut-offs endorsed by the Centers of Disease Control and Prevention (CDC).¹² For assessing platform effectiveness, the World Health Organization (WHO) BMI charts¹³ were used in addition to those from CDC, to enable comparison of results with studies using both approaches.

Usability assessment. NSUQ included a user profile evaluation section (internet access, amount of internet daily use, computer availability) and a second section regarding the evaluation of the usability perception (27 independent items, scored 1-5; the higher the index, the better the usability). The questionnaire was available on the platform. Warning emails with direct links to the instrument were periodically sent in order to reduce dropout.¹⁴

Next.Step platform. Participants were invited to get restricted access to the e-therapeutic platform (Next.Step). In addition to the platform manager (nurse), the program also had the direct support of an interdisciplinary team (including a pediatrician, nutritionist, exercise physiologist, and psychologist) who intervened when requested by the case manager. Intervention length was 24 weeks, based on a case management methodology.

The program was structured in ten thematic modules, with an average duration of two weeks. A fictional story has been created, developed in eight episodes. The story tells the life events of a fictional character who fights against the stigma of obesity and who decides to embark in a quest which will finally lead to a healthier and happier life, inspired by the magical book (the ‘Next.Step’). This is the motto for the platform, where the participant can explore five key areas: ‘Magical Book’ (where he will find the tasks/missions for each level, discussion forums, and the Next.Step episodes); ‘Enchanted Chest’ (area reserved for resources such as entertainment, games, quizzes, and competitions); ‘Lost Pages’ (virtual repository of educational resources on health, glossaries, multimedia library, and thematic stories); ‘Secrets’ (self-monitoring tools area, personal diary, therapeutic goals definition, and questionnaires); and ‘Help’ (where the platform credits are presented, technical support provided, and the virtual clinic of Next.Step can be found). Full presentation of the platform features can be found in detail elsewhere.¹¹

Results

Participants’ characteristics

Participants were adolescents. According to the CDC criteria,¹² 11.70% of the adolescents were overweight and 88.30% obese. As shown in Table 1, the mean BMI z-score according to WHO criteria was 2.661 (SD = 0.721). Participants were divided into two groups, and their homogeneity and equivalence was assessed by comparing the demographic, anthropometric, behavioral, clinical, socioeconomic, and ICT access variables (p > .05).

Table 1.

Variation of clinical outcomes at 24 weeks (Wilcoxon Signed Ranks test).

	T0		T2		Δ (T2-T0)
	M	SD	M	SD	M	SD	Z	p
Anthropometric variables
BMI z-score	2.613	0.625	2.530	0.809	-0.084	0.262	-2.815^a	0.005
Waist circumference percentile	91.427	2.950	90.151	2.729	-1.276	2.197	-4.367^a	0.000
Hip circumference (cm)	108.938	7.440	10.763	6.373	1.824	2.329	-4.701^a	0.000
Behavioral variables
Weekly physical activity (h/w)	4.363	3.141	3.939	2.459	-0.424	1.893	-1.435	0.151
Weekly Screen time (h/w)	21.77	9.598	19.802	10.759	-1.968	9.169	-2.528^a	0.011
Weight loss motivation	3.777	0.787	3.867	0.587	0.089	0.823	-0.975	0.329

BMI, body mass index; h/w, hours per week; T0, baseline evaluation; T2, post-intervention evaluation; Δ, change; M, mean; SD, standard deviation; Z, Wilcoxon signed ranks test statistic.

a

p < 0.01.

Satisfaction assessment (usability)

The assessment of Next.Step participants’ satisfaction is a crucial step in the evaluation of its usability. Table 2 presents the descriptive results of the NSUQ items regarding satisfaction perception. All items had an average score higher than three (between 3.250 and 3.875), reflecting a clearly positive perception of satisfaction.

Table 2.

Descriptive statistics of Next.Step Usability Questionnaire (NSUQ) items and Spearman correlation with the number of Next.Step consulted resources.

Item description					Consulted resources
		n	M	SD	rs	p
Satisfaction perception
1	Like using the platform	17	3.529	1.068	0.593^a	0.025
5	The platform is nice	17	3.471	1.179	0.321	0.263
6	The platform is stimulating	17	3.353	0.996	0.581^a	0.029
16	The platform is intuitive	16	3.250	1.125	0.264	0.383
15	Navigation speed	16	3.875	1.088	0.743^b	0.004
21	Time spent was worthwhile	15	3.267	1.033	0.668^a	0.017
Efficiency perception
2	Ease of use	17	4.059	1.144	0.083	0.779
3	Ease of information access	17	3.706	1.160	0.547^a	0.043
7	Ease to login	16	4.125	1.088	0.773^b	0.002
9	Ease of access to messages	14	3.786	0.893	0.200	0.555
10	Ease to download documents	14	3.571	1.158	0.167	0.624
11	Ease to communicate with health professionals	16	4.125	0.885	0.529	0.063
13	Platform availability	16	4.125	1.088	0.386	0.192
14	Platform accessibility	16	3.813	1.276	0.618^a	0.024
Effectiveness perception
17	Usefulness to treatment	16	3.625	1.258	0.480	0.097
18	Information relevance for treatment	16	4.125	1.025	0.773^b	0.002
19	Improved ability to meet treatment	17	3.294	1.312	0.384	0.175
20	Increased motivation for treatment	16	3.250	1.065	0.314	0.296
23	Motivation to discussion with colleagues about treatment	14	2.786	1.251	-0.410	0.211
24	Motivation to discussion with health professionals about treatment	14	3.143	1.167	0.038	0.912

n, number of cases; M, mean; SD, standard deviation; r_s, Spearman's correlation coefficient.

Note: items scored 1-5 (the higher the index, the better the usability).

a

p < 0.05.

b

p < 0.01.

The correlation between the satisfaction perception and the Next.Step utilization was further analyzed, and it was observed that the higher the number of consulted resources, the greater the tendency to enjoy the platform (r_s = 0.593, p = 0.025), consider it exciting (r_s = 0.581, p = 0.029) and fast (r_s = 0.743, p = 0.004), and consider the time spent in Next.Step useful (r_s = 0.668, p = 0.017).

Efficiency assessment (usability)

The second dimension of Next.Step usability refers to the efficiency indicators. The overall results of the recorded activity in Next.Step during the intervention period were analyzed. There was a greater number of participations in the “Magic Book” and “Secrets” areas, especially in the forums (2,117 views), in the wall of fame/score (388 views), and in the self-monitoring tools (328 views). The resources with lower participation were the platform games (92 views). Participants’ adherence decreased over time, with a slight increase in the third and fifth months.

Table 3 shows the descriptive results of the Next.Step efficiency indicators, regarding the recorded activity and the participants’ scores. Only 26 out of the 48 adolescents assigned to the study accessed Next.Step (54.17%), and only 23 consulted at least one feature of the platform (47.92%). Among the active participants, the average number of logins during the six months was 10.538 (SD = 18.552). The passive participation in forums had an average of 33.455 (SD = 47.032) views. The mean task-completion rate was 14.55% (SD = 18.853). The mean overall Next.Step score per participant was 63.305 (SD = 100.588).

Table 3.

Descriptive statistics of Next.Step efficiency indicators, regarding the recorded activity and the participants’ scores.

Next.Step efficiency indicators	n	M	SD
Recorded activity
Logins	25	10.538	18.552
Consulted resources	23	8.087	9.115
Active participation in forums	11	4.727	6.857
Passive participation in forums	22	33.455	47.032
Task-completion rate	18	14.551	18.853
Personal diary entries	5	2.200	1.304
Setting personal goals	8	1.250	0.463
Self-monitoring	8	6.750	5.312
Scores
“Magic Book” total	18	17.440	24.492
“Secrets” total	11	7.636	7.393
“Next.Step” total	26	63.305	100.588

n, number of cases; M, mean; SD, standard deviation.

The analysis of the descriptive results of the items related to NSUQ efficiency perception. The average score was higher than 3 (between 3.571 and 4.125), reflecting a very positive perception of efficiency (Table 2). The correlation analysis between the efficiency perception and the Next.Step utilization demonstrated that the higher the number of consulted resources, the greater the tendency to consider the access to information easier (r_s = 0.547, p = 0.043), easier to login (r_s = 0.773, p = 0.002), and easier to access the platform (r_s = 0.618, p = 0.024).

Effectiveness assessment (usability)

The effectiveness assessment is a key aspect of platform usability. Next.Step provides its users a self-monitoring tool for weight, nutrition and physical activity but this tool was only used by eight adolescents, registering an average variation (final evaluation - baseline) of -1.684 kg (SD = 2.372) in weight,.250 (SD = 0.528) in nutrition and.450 (SD = 0.691) in physical activity. Despite this favorable trend in all these three parameters, the Wilcoxon signed ranks test showed that the differences were not statistically significant (p > 0.05).

The users’ perception of effectiveness was assessed through six NSUQ items (Table 2). Only the item 23 (Motivation to discuss/share with platform peers) had an average score smaller than 3 (2.786 ± 1.251), with all the other items rating above (3.143 to 4.125), showing a positive perception of the platform effectiveness. Correlating these items with the number of consulted resources, it appears that the greater the use of the platform, the greater the tendency to consider the information pertinent to treatment (r_s = 0.773, p = 0.002).

At the end of the trial period (24 weeks), all participants underwent an objective re-assessment of anthropometric and behavioral variables. A significant reduction (p < 0.05) in the BMI z-score, waist circumference percentile, hip circumference, and weekly screen time was observed, as demonstrated in Table 1.

Discussion

The current study sought to evaluate the usability of the Next.Step platform through the users’ perception, as well as to analyze the platform performance and utilization indicators.

The users’ perception of satisfaction with the Next.Step was clearly positive. As expected, the higher the number of consulted resources, the greater the tendency to enjoy the platform, to consider it exciting and fast, and to consider that the time spent in Next.Step was useful. These findings are consistent with the ISO/IEC 9126-1 and previous studies, where it is clearly stated that online system users’ satisfaction is crucial to its success and to increase adherence to these tools.^15–18 Currently there is a growing concern with the user satisfaction and the quality of the interfaces between the different health services actors (both health professionals and patients).

The second usability domain analyzed was the efficiency. In this context, it should be noted that the Next.Step participants’ adherence decreased over time, with a slight increase in the third and fifth month. However, it should be noted that it was not possible to record the activity on external resources linked to the platform, as some videos, games, and online tools. Only 54.17% of the adolescents assigned to the study accessed the Next.Step. The high dropout rates of adolescents in online learning environments (e-learning) had already been identified by Nistor & Neubauer.¹⁹ Christensen et al.¹⁴ also reported that the high dropout rates are a common feature of online clinical trials, reaching 50%. It is noteworthy that the Next.Step adolescent participants are severely overweight, low motivated and, in most cases, with a clinical history with several weight loss failed attempts since an early age. These may be some of the reasons that contributed to the high dropout rate.

The efficiency perception was very positive; the higher the number of consulted resources, the higher the tendency of the participants to consider the platform easier to access. The development of more efficient communication channels between the individual and the health professionals has been underlined by different authors,²⁰ and it is an important requisite for a more active role in one's own healthcare. The growing dissemination of ICT may contribute to raising the population health standards through a more effective provision of care. For Garrido et al.,²¹ healthcare has benefited from the constant developments and technological innovations in life sciences. Due to a widespread ICT access, health information can be perceived as a gain in healthcare autonomy.

The third usability domain under analysis was effectiveness perception, namely the users’ ability to complete the tasks through the system and the quality of the results.⁷ The available weight, nutrition, and physical activity self-monitoring tool was only used by eight teenagers, who presented a clearly positive but not statistically significant progress. This non-significance may be due to the reduced number of valid answers.

The users’ perception regarding the platform effectiveness was in general positive, with the exception of the item on the motivation to discuss/share with platform peers. A possible explanation for this result might be that overweight adolescents, even “protected” by the anonymity provided by the virtual environment, still present limited interaction with their peers on sensitive topics. Despite these constraints, in the authors’ opinion, peer support is one of the features of the Next.Step platform with greater potential. Peer support has been considered by multiple authors^22–24 as crucial in intervention programs for obese adolescents.

It was also found that the greater the use of the platform, the greater the tendency to consider the information pertinent to the treatment. These data are in agreement with the conclusions of Lauterbach,³ who found that 57% of individuals seeking health information on the internet consider that e-health information influences their treatment options. He underlines that this does not mean that physicians are being replaced. In fact, 53% of the subjects believe that their Internet research led them to request more information from their physician or even make an appointment. In the same direction, Andreassen et al.² found that the use of e-health information appears to complement and not replace other health services.

One of the most important findings of this study was the significant reduction in some objective anthropometric and behavioral parameters, such as BMI z-score, waist circumference percentile, hip circumference, and weekly screen time. These results are in line with recent systematic reviews on e-health interventions for prevention/treatment of overweight children/adolescents.^5,25

The e-therapy was employed as an add-on and not as a substitute to the standard face-to-face intervention with the healthcare professional, based on the meta-analysis of Reed et al.,²⁶ where they presented very relevant findings: individuals who received the intervention as a complement to the standard intervention lost significantly more weight than those for whom the intervention has been a substitute. For Baulch et al.,²⁷ although the face-to-face modalities may be preferred, the high prevalence rates of obesity²⁸ suggest the urgency of expanding the treatment options and reducing the associated costs. For all these authors, the internet constitutes an innovative and effective alternative or complement to the face-to-face programs.²⁹

The usability assessment is essential, although as consequence of the distinct/specific contexts, it is difficult to make comparisons between the different systems, and potentially misleading to generalize design features. The only exception may lay in the area of subjective usability, usually assed using questionnaires and attitude scales.¹⁸

Future research should focus on additional strategies to increase participants’ adherence and alternative recruitments methods.³⁰

Funding

This work was funded by Fundação para a Ciência e a Tecnologia (PTDC/DTP-PIC/0769/2012) and supported by the Polytechnic Institute of Leiria, Portugal and by the Department of Pediatrics at Hospital de Santa Maria, Lisbon, Portugal.

Conflicts of interest

The authors declare no conflicts of interest.

Acknowledgements

This work was funded by Fundação para a Ciência e a Tecnologia (PTDC/DTP-PIC/0769/2012) and supported by the Polytechnic Institute of Leiria, Portugal and the Department of Pediatrics at Hospital de Santa Maria, Lisbon, Portugal. [Sugiro excluir, informação já dada na seção “Funding”] The authors gratefully acknowledge the clinical staff of the Pediatric Obesity Clinic and the “Unidade de Ensino à Distância” for their dedication. They would also like to thank all the adolescents and parents for their participation and collaboration.