Creating, validating, and starting all over: Survey production

Reading Time: 3 minutes
For the last year, I have been involved with the Digitally Literate Project with Drs. Michelle Schira Hagerman and W. Ian O’Byrne. This has been a fantastic learning experience as I have been able to explore so many facets of getting a large scale (international) research project off the ground, and sustaining it. When I was first brought on to the project, I was tasked with helping clean up the survey and get it ready for validation, then getting it out to the target audience (educators from around the world).

​I have written a more in-depth blog post on the validation process on the research website (find it here). However, reflecting on this post along with my experience conducting my Master’s thesis, I have come up with a few lessons learned.

1. Have a plan & be specific. It can be very easy to run wild with all the data you could collect. Your survey may end up being hundreds and hundreds of questions long if we were really able to ask all the questions we thought might be interesting (make note of these questions, perhaps you have another interesting research project on your hands!). My lesson learned on this project, as well as my own Master’s thesis project would be to define the constructs you want to measure. What constructs are important to your research? What will give you the full picture to answer your research questions? Review existing measures. What is being asked? Does it capture what you are after? What is missing?
2. Talk to people. Talk to experts. Talk to your colleagues. Talk to practitioners. Talk to your friends. Talk to your parents. Talk to your cat. Talk to anyone who will listen. This can (and should be) before, during and after creating and validating your survey. Getting multiple perspectives of what’s important to capture and how to capture it.
3. Flow & Design. Remember that there are people on the other end that don’t necessarily see your end goal when completing the survey. In my undergrad, I took one course on Human Factors and Human-Computer Interactions. My biggest take away from this was making things easy for people to use. Things need to have a logical flow and need to be intuitive to use. If you’ve outlined your constructs the flow should be easier to create, as like questions will be together. Additionally, think about how your questions will be displayed. Will drop down menus, or a check box be easier to use for the end user of the survey?
4. Be humble. It is very likely that you will think your survey is great and people are going to flock to respond. However, reality may hit. Recruitment can be difficult, and people get bored (this is where design is important and being concise with your questions). You may need to reevaluate your survey even after you have given it wings and thrown it out into the world. Could you be more concise? What items could you cut and still be able to answer your research questions? How are people responding to your items already? You may be able to shorten your survey or revise it. If you can boast a shorter completion time you may get a little more interest.

Perhaps the fifth lesson should be: Keep at it! But, I think that is just a general philosophy to have within research. Just in writing this post my brain has snowballed on to other things to talk about regarding survey development and research development in general. So expect to see more posts in the future elaborating on the process after I reflect some more and make sense of the jumble of thoughts in my head.